Construction glue is one of the strongest adhesives there is -- perfect for permanently securing a panel to a brick wall. When "permanent" ends up not quite meaning "forever," however, and you decide to remove the panel and restore the brick to its pre-glued condition, you have your work cut out for you. Softening the adhesive makes it easier to remove, but you'll have to do some painstaking cutting and scraping to get it off.
Construction adhesives (construction silicone/construction sealant/construction glue) come in a variety of formulas, and what differentiates them from most adhesives is their bulk. A synthetic rubber, latex or polyurethane paste forms the bulk, and a curing process gives the adhesive its strength. Construction adhesive typically lasts as long as the surfaces it joins and can have a shear strength of around 800 psi. It is resistant to water and most chemicals after it cures. All of these qualities make it a superior adhesive, but they work against you when you have to remove it.
Because construction adhesive is so strong, surfaces that you try to pull apart will probably break before they separate. This means that if you try to pull paneling off of brick, there's a good chance that you'll damage the brick. Even if you could get between the glued surfaces, you can't dissolve the glue, but you have an ace in the hole; heat softens construction adhesive. Running a hair dryer set to maximum heat over the material attached to the brick should soften the glue enough for you to pull it away without damaging the brick.
After you've separated the glued surfaces, you'll be left with blotches of dried adhesive stuck to the brick surface. You can use one of two methods to soften it enough to remove it. The first is to continue heating it. It will remain soft enough to remove while it's hot, but you'll have to continue applying heat to keep it that way. A second way to soften the glue is to rub petroleum jelly or mineral spirits onto it. It may take several days for either of these substances to soften the glue, however, and you may need multiple applications.
The best way to remove the emulsified glue (construction silicone/construction sealant/construction glue) is to saw it with thin 18- to 24-gauge wire. After cutting a length of wire about 3 feet long, you wrap it around your gloved fingers, work it under the glue and use a sawing motion to separate the glue from the brick. You'll get most of the bulk off this way, but you'll still have to scrape the rest with a wire brush or, if it's lodged in deep crevices, with an awl or chisel. The heat of scraping may be enough to keep the material soft, but if not, you can always apply more petroleum jelly or mineral spirits.