In the pre-War Japanese political system, people tend to think that the Emperor possessed the supreme power in the center of the Japanese political machinery. However, it is much more realistic to assume, that the Emperor was in actuality nothing but "the eye of a typhoon" - i.e. great void.

Similar to the Emperor himself his palace appears to be a great void in the turbulence of Tokyo. I presume, that foreigners who expect a big and gorgeous castle in the center of Tokyo would be disappointed, when they find only quiet woods and a few watch towers and gates as remains from the feudal period.
Jogging around the palace is extremely popular. In particular during lunch time break, you can find many businessmen running in their overalls or shorts along the moats of the palace.

However what they see on the other side of the moat is only woods and they have seldom chances to go inside and do not know what there is.

When the late Emperor Showa was seriously ill, many young people, for whom the Emperor was just a sort of ET, could only imagine him as an old wizard in the forest.
Nijubashi: the most famous view of the Imperial palace.
The bridge was built in Meiji-era.
Maruno-uchi .. central business district seen from the moat.

The Imperial palace was originally the home of Tokugawa Shoguns. The Emperor Meiji took over the palace from the 15th Shogun Yoshinobu in 1867.* The Emperor Meiji renamed the former residential city of Tokugawa Shogun and "Tokyo" ... literally "East Capital" ... was born in October 1868.
    * There hasn't been a formal decision to change the capital from Kyoto, where Emperors and Empresses lived since 794, to Tokyo. Tokyo is therefore just a tentative lodging for the Emperor. This temporary stay lasted now some 130 years, though enthusiastic Kyoto citizens still want the Emperor come back to Kyoto, the legitimate capital of Japan.
The previous name "Edo" was first mentioned in a document on 26th August 1180. Probably the name "Edo" came from the family name of "Edo Taro Shigenaga", the ruler of the port around Edo at the time. Then in 1457 "Ohta Dohkan" a feudal lord built his fortress on the sea side plateau where the present palace stands. At the time there was an inlet between the fortress and the present Marunouchi. Marunouchi and Ginza area formed a peninsula and Hibiya park was under the water.

The first Shogun of Tokugawa family, Ieyasu moved his residence to Edo at the end of the 16th century and started a massive engineering work. He ordered to acclaim land, change river flows and dig canals. The downtown area was mostly acclaimed during Tokugawa period and due to the good city planning the roads in Ginza and Asakusa areas are still very orderly and make good contrast to the present high class residential area which was only farmland at the time.

A map at a time of the 12th Shogun Ieyoshi.

A contemporary road map.

The Edo Palace was constructed on the land of 360,000 square meters and the construction itself covered 43,000 square meters. It was originally crowned by a majestic 5 story tower "Tenshu-Kaku"**, symbol of power. However, after repeated fires Tokugawa Shogunate gave up the reconstruction of Tenshu-Kaku due to severe financial burden. When Emperor Meiji captured Edo Palace there was no magnificent building.
Nowadays many tall buildings were built next to the Palace in Maruno-uchi and elsewhere, and we can look down at the inside of the palace wall. But, we cannot find much to see there.

The Emperor Showa loved the flora in his palace. One day when a chamberlain told in front of the Emperor that the weeds in the Imperial Palace were to be mowed, the Emperor objected and said, each grass and plant had its own name and no grass was named "weed".
**Tenshu-kaku was first built by Nobunaga, the most prominent leader in the 16th century. He was killed by a traitor just before the unification of Japan. He was very much interested in European technology and Christianity.

We are usually not bothered by the exact meaning of "Ten-shu-kaku". According to common understanding it means "Sky-guard-tower".

However, there is a persistent view that "Tenshu" means Jesus as the Jesuit translation of Jesus into Japanese sounded also "Tenshu".
Taking into consideration the intimate relations between Nobunaga and Jesuit monks, it is not so absurd to assume that "Tenshu-kaku" was conceived to worship Jesus and there was a statue of Jesus on the highest story, to which only a limited number of people were admissible.

Blueprint for a Tenshu-kaku of Edo Palace