How Dim Sum Legend Koi Palace Preserves the Legacy of Cantonese Banquet Culture

A chef waves a suckling pig over a charcoal fire, his wrist turning and twisting until the pig’s skin is evenly browned and perfectly crisp. ​​Children clamor over bamboo steamer baskets in the same pagoda-inspired banquet hall where their parents celebrated their wedding years ago. At first, Daly City’s Koi Palace appears unchanged by the last 25 years, a stronghold of traditional Cantonese cuisine. Then, a delivery arrives from the restaurant’s commissary kitchen. The staff unloads xiao long bao dyed five different colors, and sorts through sauce bases and stocks that will glaze Iberico pork char siu and mapo tofu mixed with plant-based meat at Koi Palace’s newer sibling restaurants, Dragon Beaux, Palette Tea House, and Palette Tea Garden.

When brothers Ronny and Willy Ng opened Koi Palace in 1996, the family had already opened Wok Shop Cafe and Happy Valley, which had three locations, in San Francisco. Ronny supplied culinary knowledge from his time cooking in Hong Kong, and Willy focused on the front of house and business operations. (To this day, Ronny prefers staying in the kitchen and letting Willy handle any interviews.) The two restaurant concepts mainly offered white clientele a mismatched cuisine Willy Ng describes as “Szechuan, Hunan, chop suey,” although Happy Valley was one of the first restaurants in San Francisco to offer hot pot. The family sold their stake in Wok Shop Cafe in 1990, but the restaurant still stands on Sutter Street. When they sold Happy Valley at the beginning of 1996, Willy thought his time in the restaurant business might be over.

Then, a property manager offered a space that could house a 450-seat dining room and was easily accessible from I-280. Willy and Ronny decided to take on an ambitious challenge­: opening a traditional Cantonese seafood restaurant that would serve Chinese Americans moving into the Bay Area’s rapidly developing and diversifying suburbs. More than 25 years later, Koi Palace Daly City is the restaurant group’s oldest property, a place where servers can split a whole fish among a table of eight with just one hand and wok cooks have committed hundreds of off-menu dishes to memory. But as the years have gone by, both the restaurant’s loyal customers and employees have grown older. Willy Ng worries that younger customers might feel uncomfortable in Koi Palace if they’re unfamiliar with Cantonese cuisine and the Chinese language. “You will feel like you are a stranger … think, ‘Maybe [Koi Palace] isn’t my place,’” he says.

So Willy Ng now faces a new challenge: making Cantonese cuisine that’s accessible to new generations of clientele, and reproducible without a classically trained kitchen staff. His solution? Sleek restaurants serving multicolored dumplings and a behind-the-scenes commissary kitchen powered by machinery and a staff of 60.

Koi Palace’s Chinese name, Lei Yue Mun (鯉魚門), references an area in Hong Kong known for eateries lined with tanks of live seafood. Willy and Ronny Ng invested almost $200,000 into Koi Palace upon its opening and imported curved glass from China in order to recreate those tanks. Willy Ng makes it clear — Koi Palace isn’t just a dim sum parlor, but also a restaurant preserving the rich history of Cantonese cooking. “You know in China, the Eight Great Traditions, the eight different provincial cuisines? Cantonese [cuisine] is always number one,” he says.

Koi Palace, now with additional locations in Milpitas and Dublin under the same management team, provides a space for Cantonese Americans to uphold traditions centered around food and banquet culture. Waiters in suits and ties carry platters of hard-boiled eggs, dyed red, to families celebrating the health of a newborn; groups celebrating Chinese New Year gather around poon choi, a large bowl layered with meats, seafood, and vegetables that symbolize prosperity. The surface includes Cantonese cuisine’s most prized ingredients: abalone braised in a glossy brown sauce and prawns the size of one’s fist with impossibly bright red shells.

Most of the dining room is filled with elderly regulars, some of whom visit Koi Palace five to six times a week. They practice yum cha, a phrase that literally means “drinking tea” and refers to a common practice of gathering with friends and family to share dim sum. Willy Ng splits his time between Koi Palace and his newer restaurants but treasures the opportunity to sit down in its dining room and greet old customers. “Some [bring] their own children and grandchildren. Over 25 years, so many things happen … I always say hi, and although sometimes I cannot recognize their face or their name, I still feel warmth, closeness, and familiarity,” he says.

But Willy Ng fears younger diners, especially second-generation Asian Americans, may feel alienated by the traditions preserved at Koi Palace. Ordering in Cantonese banquet halls often requires a certain level of cultural knowledge; many of Koi Palace’s fish tank residents aren’t listed on the menu, so customers have to ask, “What seafood is fresh today?” “How much does it weigh?” Some older customers might ask to see their chosen fish splashing around in a plastic bag as proof of its freshness. Deciding the seafood’s preparation can resemble a negotiation — should crab be stir-fried with scallions and ginger, shelled and served with steamed egg whites, battered with salted egg yolks, or some combination of these off-menu methods? Ng knows his loyal customers feel at home conversing with the servers who have walked the dining room for decades. He wants to avoid making any changes that would threaten this sense of familiarity for his longtime patrons, while still offering a welcoming environment to those less familiar but willing to learn.

Meanwhile, the waning number of classically trained Cantonese chefs, coupled with the questions the pandemic has raised about demanding, often low-paying kitchen jobs, makes Ng predict that someday only a few large banquet halls will exist in the Bay Area. Referencing how Koi Palace’s lead barbecue chef has been hunching over a charcoal fire for over 25 years, Ng asks, “Who would like to learn this kind of job? If I pay you 50 percent more than your current pay, you still won’t want to do it. Without passion, no one will want to do it.”

Faced with a desire to serve a broader clientele but a dwindling number of experienced chefs, the Koi Palace restaurant group has shifted toward opening restaurants with contemporary interiors, streamlined menus, and notably Instagrammable dishes (the rainbow soup dumplings have become particularly iconic/sticky on social media). At Dragon Beaux (2015), Palette Tea House (2019), and Palette Tea Garden (2020), some differences from Koi Palace are small. The fish tanks have been branded with the name “Live Seafood Showcase,” and there’s a menu listing the seafood options with their origins. Item names are written in English, Chinese, and Pinyin (a romanization of Chinese characters for those who can speak, but not read Chinese) to account for all levels of Chinese proficiency. Other changes are more noticeable, including a cocktail program designed by bartender Carlos Yturria (the Treasury, Absinthe, and others) and staff and clientele who are far younger and more diverse than at Koi Palace.

Chefs at Dragon Beaux and the Palette restaurants showcase their creativity and skill through dishes like lobster dumplings pierced with pipettes of warmed butter that customers inject themselves. But without a large team of cooks experienced in Cantonese cuisine, many members of the kitchen staff focus on preparing only a couple of items and learning generalized forms of Chinese culinary technique.

At the same time, Willy Ng says customers’ palates are more discerning than ever because this younger clientele has grown up eating at excellent Chinese restaurants. This dilemma led him to open a commissary kitchen in Millbrae in 2011. About 60 employees work in the commissary and prepare dim sum, soup bases, and sauces that are delivered to the group’s restaurants and make life easier for the cooks. This model takes its inspiration from Ng’s visits to China and Singapore, where these wholesale kitchens are commonplace. The commissary has been integral to the restaurant group’s operations for more than a decade, and many customers have unknowingly enjoyed its products at Koi Palace, catering events at tech companies, and through online orders of frozen dim sum during the pandemic.

While the commissary is filled with machinery, operations are overseen by Koi Palace’s longest-tenured chefs. They make sure that parts of the process are done by hand, if that is the method that creates the best product. “Authentic means traditional … authentic for the Chinese is a very deep history, a very deep knowledge,” Ng says. So while mooncakes might be stamped and shaped in a $80,000 machine instead of wooden molds, salted duck egg yolks are folded in manually. Meats are marinated before roasting, and soups are made in the traditional style of double-boiling, meaning ingredients are steamed with water in an enclosed bowl to preserve their texture and flavor.

In a world where ghost kitchens and delivery services can foster concerns about the future of restaurants, Ng is excited. He sees an opportunity to carry on Koi Palace’s original goal – serving Cantonese food made with traditional techniques. Ng hopes to one day sell frozen, restaurant-grade dim sum to some of the nearly 50,000 Chinese restaurants in the United States. “My dream is to be in every restaurant. You can eat egg rolls, spring rolls, and pot stickers [in any Chinese restaurant in the United States right now]. But in the future, I hope there will be xiao long bao and siu mai in every restaurant,” he says.

Back at Koi Palace, however, Willy Ng’s goals are more modest. He plans on completing one more renovation at the Daly City restaurant but will preserve its menu, traditions, and fish tanks. When explaining his vision for Koi Palace, Ng employs two appropriately old-school Cantonese phrases. The first one translates to, “We’ll open until the staff retires.” The second, “Make guests feel at home.”