OpenAI MRC

I have added several links about MRC before, but wanted to read the OpenAI paper for myself.

0 Abstract
tail latency dominates performance of synchrounous pretraining jobs
MRC (Multipath Reliable Connection) = new RDMA-based transport and a bit of UET, packet spraying, adaptive LB based on ECN, out-of-order memory placement, select ack and packet trimming to mitigate incast. RDMA: only support write and write-with-immediate
multi-plane clos -> >100k in 2-tier, high switch radix
static source routing SRv6, bypass failures

1 Intro
solution needs: LB, incast, handle link and fabric failures gracefully
disable dynamic routing!, only static paths with SRv6

2 Multi-plane topology co-design
T0=TOR, 64x800G. Use 8x100G instead -> 8 planes of T0. TO connects to 256 NIC and 256 T1 (spines) -> total 131072 GPUs in 2-tier
many advantages: latency, more nodes reachable in 1 hope, less cost and power (2-tier only), less impact link failure
problems: survice link and nic port failures. load all network planes and LB all many paths!

2.1 MRC overview
extends RoCEv2 + UET features (multi-path)

  • each packet has the RDMA virt address and remote key -> receiving nic can write at arrival, no matter order.
  • each packet has an entropy value EV (32 bits) Sender rotates the EV for each packet -> LB (packet spraying)
  • PFC disabled (it doesnt work with so many possible paths)
  • fast selective retransmissin = Selective-ACK.
  • under incast -> packet trimming = packet is not dropped, reaches destination, destination triggers NACk = retranmission Each EV corresponds to a specific path on a specific network plane

For each EV, MRC keeps a few bits of state about path health. ECN used but disabled in the last hop to the receiver -> ECN acts as LB signaling (different

MRC senders do not coordinate when choosing their EV sets)

When packet lost, EV path is stopped then MRC sends background probes to verify, if enough probes succedd -> EV resurrected

2.2 Static SR
difficult to maintain large ECMP sets. MRC avoids failed paths, then dynamic routing re-routes -> changes ECMP -> disturb LB => disable dynamic routing
How to map EVs to network paths? In the MRC nic, at QP startup a set of EVs are chosen, bits in each EV directly embed the path choice available at each hop in the network.
SRv6: use microsegment ID (uSID), where destination ipv6 is 32b prefix followed by a seq of 16b uSIDs each corresponding to a specific switch along the path. Each swith left-shifts the uSID part of address. Switch has a static forwarding table, configured when installed, rarely changed. uN forwarding is done at line rate.
MRC packets are ipv6 in ipv6 encapsulated. outer dst = srv6 path, inner dst = dst nic

2.3 Mapping EVs to SRv6 address
with srv6, ev is not hashed by switches, but sill needs to be carried in data packets so received can echo it. SRv6 address can’t be echoed because it is erasedd by the shiffiting process during forwarding
algo mapping between EV and its SRv6 address.
*** COMPLEX need to review last part

2.4 Choosing working paths
On QP startup, MRC must select a set of EVs, each mapping to a unique path on a specific plane. EVs are equally split across the planes, then randomly selects a subset of path within each plane. No necessary to use Clustermapper to set denylists for failed T0-T1 links.
*We have to rum-up large training jobs slowly to avoid destabilizing the power grid.
On the reverse path:

Reverse path: many QPs are unidirectiona at any instant. Keep small reserve EV set for control packets, with at least one EV per plane. Each RTT that the QP is acive inbound but generates no outbound traffic, the receiver send an EV prbe packet using a randomly chosen EV. If the probe is acked the reverse EV for that plane is set to the probe’s EV. If data traffic is sent, the reverse EV is updated from data SACKs instead of needing EV probes.

3 Operations
With MRC, Most network failures dont require quick reactions from netops. Link-failures and flapping links between T0-T1 can be ignored!!! MRC spreads traffic across enough paths that when a link used by a QP flaps, only a very small number of packets per QP are lost, and the corresponding EV is removed. The missing pacet is selectvily retransm on a different path. MRC maps paths out when they drop, and only brings back when enough probes succedd over time -> less coordination with netops
Can reboot T1 switch easily. NIC-T0 and T0 switches are more problematic -> After link drop, remapping all the EVs fro the many QPs in use is not instant. MRC uses a port state bitmap in SACK packets to notify remote QP endpoints that the port is down, so they also remap their EVs to avoid the failed plane. This takes a few seconds.
CLustermapper agent running on all cluster nodes together probe every link in the network every ms!! This check NIC-T0 links and T0 switches. Agent send probes source-routed to the T0 and back to the same agent. Each agent also probes a subset of T1 switches, so all T0-T1 links are probed. ICMP probes are handled by CP, so low frequency. With SRv6, the switch treats a probe like another data traffic, by DP -> higher frequency.

4 Inter-plane Loading
Keep all planes evenly loaded <- when MRC maps an EV out of its active set, it replaces it with one from the same plane -> aovid false incast at destination

5 Experiments
flows: T0-local or cross-T1

5.1 Training Results

  • Link flaps almost no impact -> low priority at operations level
  • NIC-TO links does affect performance, but most events are transient and job recover full speed quickl.
    optical transceiver glitch in T0. NIC and Switch dont match the time when the port was down. Still job didnt crash and fully recover. This is rarer that individual link flaps.
  • T1 switch reboot: QP packet lost only at T1 failure. reboot took 2 minutes and no impact during reboot as all QP wer mapped out of the bad path. throughput largely unaffected.
  • NIC transceiver are SPOF, they flap, but rarely. A different optical design might avoid this issue.

5.2 Testbed results (MRC in different NICs: mellanox cx-8, amd pollara, broadcom thor ultra)

5.2.1 p2p communicatio performance – cluster B
cx-8. t0-local and cross-t1 = aprox 770Gbps (96% theorical peak bw)
t0-local 5.09 us and cross-t1 6.54 us (extra switch hop). High latency effect only short-message, negligible for large

5.2.2 MRC response to link down and flap events – Cluster B
less 1 sec interruptions during failure detection . MRC detecs link down event and rebalances EV across remaining planes and updates link0state btmap in SACK packets to notify remote endpoint.
T0-T1 failures have a substantially smaller impact than nic-t0 failures, because MRC sprays packets from each QP across a large number of paths.

5.2.3 MRC behavior with T0/T1 switch failured – cluster B
T0 switch failure -> bw drops 100Gps. MRC failure handling is driven by end2end path availability rather than specific location or type of underlying fabric fault -> swithc-level failures manifest a predictable reduction of usable path diversity, preserve app progress and maintein stable throughpput
T1 switch failure -> no degradation is observer. QP are sprayed across a large number or paths and sufficient alternative EVs remain available to sustain lost of capacity.

5.2.4 Robustness to path-level packet loss
Introducing errors in a EV, caused it to get inactivated and activate another “spare” EV

5.2.5 Load balancing accross EVs
Upon detection of congestion via ECN, MRC triggers traffic distribution to rebalance load across the available EVs. MRC can effectively rebalance traffic across EVs in repspomse to dynamic load changes.

5.2.6 NCCL collective execution at scale
sendrecv benchmark: 92GBps in 42k GPUs

5.2.7 Comparison with RoCE
MRC 1QP sparying across 256 paths achieves better performanc that RoCE with 16 QPs
At 1% loss, RoCE is not usable. The same for MRC because the induced lost affected all planes.
For all message sizes MRC outperforms RoCE but the differnce is greater in the regime that is bw-bound (larger message size)

At smaller message size, the collective is more latency-bound
At bigger message size, the collective is more bandwidth-bound

5.2.8 Collateral Damange (Incast)

Configuring DCQCN properlyis very hard because it is traffic pattern specific to the point some hyperscalers have disabled it.

6 Related Work
Load balancing effectiveness is defined by distribution granularity: per flow, per subflow and per packet.
RocE2 uses per-flow ECMP. Per-packet load balancing (or packet spraying) has been proposed for CLOS networks to reduce amounf of state per (virtual) path -> use ECN or similar

Multipath TCP: must keep state per subflow. Used by Google Falcon to replace RoCE2 in Nic

MRC changes RoCE2 to enable packet-level load balancing, selective retrans and improve reliability in multi-plae networks: its target deployment is best-effort (lossuy) networks that support packet trimming.

MRC is similar to UltraEthernetTransport (to replace RoCE2)

7 Conclusion
MRC is designed to LB a multi-plane network by spraying each QP across all planes and many paths in each plane, performing fine-grain active LB and routing around failures. Disabled dynamic routing, use SRv6 source routing with static routes in the switch

Caliban’s War

This is the second part of the Expanse series.

Now I understand the name of Rocinante for the spaceship. The guys are like Don Quijote trying to beat giants (or space monsters)

The book is engaging and I devour it in a couple of days but I felt it was the same setup as the first one.

I will read the next one, it is in my to-read list.

The code of the extraordinary mind

I am not keen of hypes but I could find some truths in this book.

One of the first things is to question the Brules = Bullshit rules. Those are the rules passed on us by society, family, religion, etc. I tell myself that I need that certification, I need to save, I need to know X, then Y, Z. And I know some are excuses…

I struggle with the concept of working hard and enjoy life. The ideal is to enjoy your job, but I dont see myself enjoying my job working 12h a day. I tell myself I would like to work in a bakery for 1 year, like a sabatical. No balls yet to do that. I am waiting for the moment…

The book recommends to take this Lifebook Assessement:

  • Your love relationship
  • Your friendship
  • Your finances
  • Your quality of life (materialistic things)
  • Your health and fitness
  • Your intellectual life
  • Your emotional life
  • Your spiritual life
  • Your career
  • Your character
  • Your family and parenting life
  • Your life vision

I find it useful, because I can see the positive things of my life and where to improve. And most of the book is around those 12 points.

Related to the b-rules, we have models of reality embedded deep within that we are not really aware of (what you don’t know you believe!) Example: you are enough and you have nothing to prove. This means you need to rewrite your models of reality (ie: you are attractive, you are good enough, etc). Two questions then:

  • Is my model of reality absolute or relative truth?
  • Does this really mean what I think it means?

Regarding setting a vision:

  • We confuse b-rules with goals: having a certain job, appearance, etc is b-rule.
  • We are notoriously bad at predicting just what we can do in a given time frame: a) we tend to overestimate what we can do in short term (1/2-1y) and underestimate what we can do in long-term (3y)

Goals are part of a vision: but some goals are part of b-rules so how to identify:

means goals (not good):

  • means goals usually have a “so” in the them: Get this job so I will be happy
  • means goals are often about meeting or conforming to b-rules: have a degree

end goals (good)

  • end goals are about following your heart
  • end goals are often feelings

Something I do every night is to gave grace/gratitude. I can’t really explain but it calms you down, it put things in perspective, and you sleep better. I trully believe that I am the luckiest person, whatever happens, period.

The next thing, it is forgiveness. I found it difficult, but makes sense, it is a way to liberate yourself. I need to do that with the gratitude.

Last thing, is “to give”: To be happy, make others happy. Who can you help if you are unhappy?

Be unfuckwithable: Extraordinary minds do not need to seek validation from outside opinion or through the attainment of goals. Instead, they are truly at peace with themselves and the world around them. They live fearlessly, immune to criticism or praise and fueled by their own inner happiness and self-love.

exercises:

  • The person in the mirror (self-love)
  • Self-gratitude (for appreciating yourself)
  • Become present (to remove sudden fear and anxiety)

The authors meditation six pack, daily, no max of 15 minutes.

  • Compassion
  • Gratitude
  • Forgiveness
  • Future dreams
  • Perfect day
  • The Blessing

In general, I can find things that resonate and I need to put in practice. I should re-read again at some point

Classic Brownie

This was my first “brownie” many years ago, and shame on me, I haven’t tried for several years. So this deserves to be here. Super easy to do.

Ingredients

  • 300g chocolate (I used 90% dark)
  • 300g sugar
  • 200g butter
  • 6 medium eggs
  • 90g all-purpose flour
  • Tahini (extra)

Process:

  • Pre-heat oven at 160C
  • Mix eggs and sugar, very well. Use a handmixer if you can.
  • In a low-medium heat pan, melt butter and chocolate.
  • Add choco-mix to the egg-mix and mix again very well, use a wooden spoon.
  • Add the flour and mix well again
  • Wet a sheet of baking paper and put in the your baking mold.
  • Pour the mix in the mold, be sure it is even
  • Extra: pour the tahini (it has to be a bit runny) in the top and with a wooden stick or similar mix it in the surface like below
  • Bake for 20m. This is critical, dont over bake it! If it is a bit runny in the middle is fine. If it is dry, then you over did it.

My result. It was really good!

Let is cool down for a couple of hours before cutting.

Dating, Backwards, Adversity, SR-IOV, Meta EBB, opensm, IB EEPROM, heartbeats

Dating summary: Really good all points, most of them I feel them or experienced.

Backwards Law: Emotions are weather, you can’t control them. You can control what you do with the emotions

You never know who you are until you have adversity in life.

SR-IOV:

SR-IOV (Single Root I/O Virtualization) is an extension to the PCI Express specification that lets a single physical device, like a network adapter, present itself as multiple separate virtual devices. Each virtual machine gets its own direct access to the hardware, bypassing the hypervisor’s software layer and delivering network performance close to what you’d get on bare metal. It was designed by the PCI-SIG standards body specifically to solve the performance bottleneck that virtualization creates when many VMs share one network card.

From this article about Meta Superintelligence, came across this link about Meta backbone designs at scale for EBB

opensm: I can’t find many blog posts with real hands-on about IB things. And this is a good one. It doesnt just show you the configuration, but the problems you can find. It seems it is better to use the Mellanox’s opensm

  • systemctl status opensm –no-pager
  • sudo sminfo
  • sudo ibnetdiscover
  • sudo saquery NR | grep -E “lid|NodeDescription”
  • sudo tail -f /var/log/opensm.log

IB cable EEPROM: Interesting how critical can be having the correct cable from the EEPROM.

 The label on a cable is marketing. The EEPROM inside it is a contract, and InfiniBand only honors contracts.

  • mstlink -d –show_module

Veritasium: 1 billion heartbeats

Very Bad People

Very good book about the origin of Global Witness.

It is really humbling how three people started all this on their own. We capable of horrible things but as well, amazing things. So maybe your effort is just a drop of water but it can add up.

I had no idea about the deforestation mafia of Red Kehmr in Cambodia and how that financed the war. And worst, how both sides were profiting from it. As always, the population is the one that pays the consequences.

That follows the diamonds from Angola, the petrol from Guinea, the wars of Siorra Leone and Liberia, etc.

I think of several countries as corrupted, but as the book shows, it is the first work with the financial engineering and PR/Law firms that allow all the mone to be moved away from those countries

If we dont allow those people to buy property, ferraris, etc it would be less convenient…. and make all the companies accountable.

At the end of the day, while fiscal paradise exists, this will continue, but at least there are people like GW that will pull the carpet and show the ugliness…

The Courage To Be Disliked

I have been quite surprised by this book. It is based on the psychology work from Alfred Adler. And i take it more as philosophy than anything else.

It is based on teleology: the study of the purpose of a given phenomenon, instead of its cause (that is aetiology). We determine our own lives according to the meaning we give to those past experiences. So it negates the influence of the past and traumas. The important thing is not what one is born with, but what use one makes of that equipment. We need the courage to be happy, because that needs change (the lifestyle), and it is scary. As well, this makes you to focus in the present.

All problems are interpersonal relationship problems. Personally, I feel my goal is not to be hurt in relationships with other people. But it is impossible not to get hurt (or hurt somebody).

Feelings of inferiority are subjective assumptions (and excuses), and those we can change it. Boasting is an inverted feeling of inferiority. If one really has confidence, one doesnt need to boast.

Life is not a competition (winner vs looser) and this applies to Relationships too. This make you see people as comrades. Only compare with your ideal self. You are the only one worrying about your appearance. When there is competition, there is a power struggle. Avoid the conflict as soon as possible, dont answer the action with a reaction (this is not admitting defeat though), because this evolves to a revenge.

Our objectives are: self-reliant (I have the ability) and live in harmony with society (people are my comrades). Our life tasks are: tasks of work, frienship and love

Life lie = I am making up flaws in other people just so that I can avoid my life tasks, and more, I can avoid interpersonal relationships -> courage.

As we have our tasks, the other have their tasks. As we focus in our task, looking for recognition is debilitating, it creates a dependency, a vertical relationship (you want horizontal relationship). Do not live to satisfy the expectations of others. This means freedom. Freedom to be disliked by other people. The same way, you dont have to praise or debuke. Saying “Thank you” is good enough

This leads to the goal of interpersonal relationships, that is the feeling of community. This is acquired by your own efforts, active commitment.

Friction, Morning Routine, RoCe Meta Paper, AWS RNG, MRC, Slurm, Rail-Optimize, 800VDC, Phyllo, Approach

Manson AI: You need friction

AI Agent: narrow focus: goal, proof, steps

Bear Grylls’ Morning Routine: Cold (never get used to), bared foot, strength training, 30 minutes

RoCE networks for distributed AI training at scale: I have managed to read the paper ! Although in the AI word, two years is an eternity, I think it is still interesting.

1) Network Topology: backend network only for GPUs (RDMA nics), non-blocking. Frontend network: data ingestion, checkpointing, logging.

Pod = AI zone
 leaf = RTSW, DAC cables, shallow buffer
 spine = CTSW, deep buffers. fiber between leaf-spine.
SuperSpine = ATSW, oversubscribed, connect AI zones

intra-node -> nvlink
ROCE: cpu offloading, ethernet (standard)

collective communication library serves as the sw abstraction between training workloads and the NIC
                                 schedules verbs calls over QP (Queue Pairs)
 parallelism strategy determines collective: allreduce, allgather, alltoall
 choice logical topology:

------------------

2) Routing: work load. low entropy flows (few flows) -> ECMP bad (5-tuple udp: src/dst ip, src/dst port, protocol), burstiness, elephant flows
--

 RTSW uplinks 1:2 under-subscribed! -> expensive (short-term)
 1) QP scaling: use destination QP of Roce packet using the UDF capability in switch to increase entropy -> Enhanced ECMP -> short-term
 2) Central TE controller -> long-term: CP real-time topology end-to-end cluster, 
                                        flow matrix (flow bps) + CSPF (constrained SPF)
                                        write in switches dataplane
                                     DP: TE overrides default BGP routing policy in leaf. Use Exact Match table.
                           Not good with multiple link failures. Doesnt scale 
 3) Flowlet switching: try to improve 1 and 2. hw assistant schema. put packets in different ports in ECMP
     out-of-order: move packets only after 1/2 RTT
     load-aware path assignment: better than TE

------------------
                     
3) Transport: congestion management. Start with DCQCN. packet drops on ACK/NACK can cause prolonged Local ACK timeout (LAT)
--
 Tuning DCQCN not great (strict ECN -> minimize PFC (can lead to head-of-line blocking)

 200G, we stayed with relaxed ECN marking, allowing for buffer build up in the CTSW, while keeping default DCQCN settings.
 400G We proceeded without DCQCN. just PFC for flow control
 re-design collective library: two-stage copy

------------------

4) Operations:
 Change QoS priority of Clear to Send (CTS) messages. In RTSW ASIC, modify dsCP marking for ACK  messages
 Tuning VOQ in CTSW
 obeservability: OOS: out of seq.
                 Link flaps
                 Local ACK timeouts (LAT)
                 PFC watchdog: catch any long-duration PFC pause (>200ms)
                 buffer utilization RTSW
                 reachibility (pings)
                 constant latency monitoring loaded and unloaded (catch regressions)
                 base lines!!!

Perplexity: Hosting Qwen on Blackwel:

AWS RNG – Random Graph Network: The paper is totally out of my space, but the concept looks brutal. With an operations hat, how you troubleshot it? (ping, traceroute, link congestion, data flows patterns, etc)

MRC1 and MRC2 (OCI): Why we need planes (breakouts) and not just a big plane.

As SerDes speeds continue increasing, every microsecond of congestion creates much larger pressure inside the fabric. A 100G transport domain may be manageable. A 400G domain amplifies the same congestion into roughly 4x pressure. An 800G domain, and eventually a 1.6T domain, becomes much harder to coordinate.

This pressure appears as larger switch buffer requirements, larger congestion domains, harder retransmission coordination, larger cache pressure, larger synchronization storms, and harder thermal and power scaling inside ASICs.

At hyperscale, switch ASIC cache and transport coordination become fundamental scaling bottlenecks. Increasing switch buffer size is extremely difficult: high-speed SRAM is expensive, larger cache arrays consume significant power, thermal density rises quickly, die area scaling becomes inefficient, and routing complexity increases dramatically.

Splitting transport into many smaller lanes naturally reduces these pressures. Reliability improvements then emerge as a byproduct, because congestion, retransmission, and buffering become more distributed.

THE QUESTION: which breakout keeps the fabric at the shallowest practical Clos depth while keeping plane count and operations manageable? -> less hops, less switches, less latency

Slurm: I like the “Slurm vs. Kubernetes”

Slurm Workload Manager (short for Simple Linux Utility for Resource Management) has become a cornerstone of large-scale computing. Originally created in the early 2000s to support large-scale high-performance computing (HPC) environments, Slurm is now widely recognized as the de facto scheduler for HPC clusters. Today, it orchestrates jobs across thousands of servers and GPUs in some of the world’s most advanced computing environments. 

Interview Question: 512 GPU, non-blocking (full bisection) and 2xUFM! I really liked this. I think for once I understand the rail-optimize (fat-tree = leaf-spine). Just break one leaf-spine link, beautiful!!!

800VDC: Next step in electrical infra in DC space.

Phyllo by hand

Approach woman: curiosity and no performance. Practice. Be at peace with uncomfortable and akwardness. Rejection as learning

Genghis Khan

Very interesting book. In Western Work we know a lot about Roman Empire, Alexander the Great, etc. But we dont look very often to Asia. And Gengish Khan and the following Mongol empires shaped much of the world society on the time and until know.

He was focused in meritocracy. As part of his war strategy, it was the elimination of the aristocracy of the conquered land. Very strong focus in integration. They never imposed their culture, they had full freedom for religious belief. They were brutal in war but never cruel. Torture was common in Europe and other empires, for them, it was against their belief.

They had a very clear war strategy: light travel, fast striking. They had few luggage and basic diet. They cleared the path for their horses for advancing and returning. So they destroyed and agriculture over their conquering paths.

It is interesting how Genghis Khan crumbled after his death because he didnt manage this family properly. But still, the new kingdoms kept a balance for a long time.

And something that reminded me to Rome, they had to keep expanding the empire just to keep happy the capital…. They introduced the paper money and women ruled when the men were fighting… and their campaigns lasted years!

Trade was critical for Mongols. They reached Hungary and the Balkans. They trade slaves with Venice and Genoa.

The climate was critical for their success, when the weather became warmer, their pastures were less productive in Mongolia, they had less horses, so the base of their strength was tilted.

They were master of propaganda, to spread fear so they conquering was easier. The empire was based on good army, good propaganda and good administration (just think of the sear size of the empire). They founded public education.

Mongols unified China. I didnt know that, they founded Beijing and started Forbidden City. They created the Chinese identity but they followed the mongol customes behind the curtines.

And the end of it was the Plague. The Plague stopped commerce and people. Without the fluid transit of people and goods, they couldnt keep it together.

And it is really shocking the bad reputation that has been written about Mongols after their incredible empire and success.

Portokalopita

This is a cake I wanted to try after I visited Greece with my friends. I never had the name of the cake, but after that holidays I did dome research and I think I found the name, portokalopita. I’ve got a receipe, but then I did nothing.

So finally, I tried. The taste is similar but the execution is not great. Mine is too runny.

Ingredients

Syrup

  • 1 1/2 cup orange juice (just squeeze it from oranges…. I didnt do it)
  • 1/2 cup water
  • 1 1/2 cup sugar
  • 1 cinnamon stick

Cake

  • 180g phyllo sheets (I think I need double)
  • 4 eggs
  • 1/2 cup sugar
  • 1 cup olive oil
  • 2 tsp vanilla extract
  • 200g yogurt
  • 2 tsp baking powder
  • orange zest from 2 medium oragnes.

Instructions

  • Preheat oven at 120C.
  • Place the phillo sheets on a tray. Cut them in slices so it easy to fit. Get them in the oven, until hard and crunchy. Turn them over when needed. Remove from oven and let is cool down
  • Make the syrup. In a small sauce pan mix the orange juice, water, sugar and cinnamon. Bring to boil, then reduce heat and simmer for 7 minutes. Set aside to cool
  • Set oven at 180C
  • In a large bowl add the eggs, sugar, oil and vanilla. Beat until frothy.
  • In a smaller bowl, mix the yogurt and baking powderand set aside for 2-3 minutes. Add the yogurt to the egg mix
  • Add the orange zest and crumbled phyllo into the egg mix gradually.
  • Grease an oven dish, and pour the cake mix. Even it out
  • Bake for 35-40 minutes or until the top is dark golden.
  • Remove from oven and make a few slashes with a knife and immediately drizzle the syrup slowly.
  • Let the cake sit for 2-3 hours.
  • Keep in the fridge for 1h before serving.

The result:

I think the syrup is too much and I my cake mix needed more phyllo

I will try this next time