From bd06577c82b7b18f902ddb06697542c5a06ade2e Mon Sep 17 00:00:00 2001 From: sethp Date: Fri, 10 May 2024 17:56:38 -0700 Subject: [PATCH 1/6] Update riscv-router.md --- riscv-router.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/riscv-router.md b/riscv-router.md index b0126ec..3986318 100644 --- a/riscv-router.md +++ b/riscv-router.md @@ -31,7 +31,38 @@ And some broader tensions that have impacts here: ### Comparison -with https://store.ui.com/us/en/pro/category/wired-edge-max-routing/products/er-x + + + + + + + + + + + + + + + + + + + + + +
+ Design GoalEdgeRouter XEdgeRouter X-SFP
Price$40$60$100
Power (max)5W5W
Power (avg)??<5W ?
ISARISC-VMIPS
Networking(2-4) 10/100/1000 RJ45 ports(3) 10/100/1000 RJ45 ports
(1) 10/100/1000 RJ45 port (Data/PoE input)
(1) 10/100/1000 RJ45 port (Data/PoE passthrough)		(5) 10/100/1000 RJ45 ports
(1) 100/1000 SFP port
PoE Output??N/APassive 12W (24V)
Software??Linux (Vyatta)
Other Hardware?? + + + + +
ProcessorDual-core 880 MHz, MIPS1004Kc
Memory256 MB DDR3 RAM
Storage256 MB NAND
+
Ambient operating temperatureAt least +10 to 40°C-10 to 45° C (14 to 113° F)
Ambient operating humidity??10 to 90% noncondensing
Certifications??CE, FCC, IC
+ ### Gigabit Ethernet From ac6e40f980a32cf1afb2d71fbfaf67cc677fe4e9 Mon Sep 17 00:00:00 2001 From: sethp Date: Sat, 11 May 2024 11:25:33 -0700 Subject: [PATCH 2/6] Add FPGA rough notes, some references Especially of interest is how "wide open" the FPGA accelerator field seems to be. --- riscv-router.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/riscv-router.md b/riscv-router.md index 3986318..8c326dc 100644 --- a/riscv-router.md +++ b/riscv-router.md @@ -84,6 +84,10 @@ What is the minimum clockspeed? Per https://en.wikipedia.org/wiki/Gigabit_Ethern * Mycelium (the small fungi filaments that connect nutrient pathways between trees and fungi) * \ +## References + +- https://zipcpu.com/blog/2023/11/25/eth10g.html +- https://www.chili-chips.xyz/open-cologne/ (note the paring of a Raspberry Pi CM4 "IO board" with a Lattice/Cologne FPGA) # software @@ -125,6 +129,8 @@ How do we measure things at 1 gigabit? # part selection / pricing +## fpgas + https://www.xilinx.com/products/boards-and-kits/arty.html ? Xilinx Spartan 7? (in the logic analyzer) @@ -137,6 +143,21 @@ This looks promising: https://www.microchip.com/en-us/products/fpgas-and-plds/fp And they start at ~$14/chip: https://www.mouser.com/c/?marcom=134980455&sort=pricing +### Rough notes + +It seems like there's a few layers to try teasing apart here: + +- Development board/kit; probably we should expect to go through a few of these, probably starting with whatever's easiest for beginners and going from there? There's a lot of details in the FPGA design space, and it's not clear how to tease them apart yet. There's also a lot of play in the FPGA-board-maker space, too: most seem to also integrate whole SDKs on top of the board on top of the FPGA itself (see "programming software" bullet below) +- "Production" chip: defining the envelope we need from the part is one thing; finding anyone brave enough to have an open FPGA core is a whole 'nother. Also, availability & price seem weirdly un-correlated to the dev boards that I've seen so far. +- "Acceleration" chips: The best way to do FPGA development seems to be on an FPGA.... but, not "directly," instead using the FPGA to coorindate with the RTL simulator. What's _that_ chip selection look like? + - Amazon will rent you an [F1 instance with a single (1) attached FPGA](https://instances.vantage.sh/aws/ec2/f1.2xlarge) for $1.6500/hr on-demand (~$0.6488/hr spot), which is about $1200 a month ($500 for spot). They've got a density problem, it seems, since the largest size is an [`f1.16xlarge`](https://instances.vantage.sh/aws/ec2/f1.16xlarge) which comes with 8 FPGAs and a whopping 976.0 GiB of RAM with 64 vCPUs; that and the dedicated 25 Gigabit networking suggest they're renting you the whole blade at that point. But: in our (very limited) experience with RTL simulation, a fairly complex model with no acceleration takes on the order of ones of CPUs and tens of _Megabytes_ of memory. Meaning, a burst-credit model for way oversubscribing the CPU and nano- or micro-scale memory (0.5 to 1 GB) might get us equivalent performance to the `f1.2xlarge` at literally 1/100th the cost of the `f1.2xlarge`. But, to sell us that, assuming their blade is already otherwise "minimally sized" for an AWS DC, they'd need to slot some 512 or 1024 FPGAs per blade. Maybe in the `f2` class, eh? + - Or, it's possible that _is_ a good deal for running a workload more than a couple hours a month, and I've wildly missed my guesses above. In which case, the FPGA itself makes up a significant proportion of the price, and the kind of hardware that people use to simulate other hardware costs on the order of tens of thousands of dollars. I sure hope not! + - But, uh, "firesim" (which is the FPGA accelerator underneath chipyard?) [supports](https://docs.fires.im/en/stable/FireSim-Basics.html#choose-your-platform-to-get-started) the Xilinx Alveo U250 (or U280, which is [discontinued](https://www.xilinx.com/products/boards-and-kits/alveo/u280.html)), [Xilinx VCU118](url), and the RHS Research Nitefury II. That's a [$10,000 "accelerator card"](https://www.mouser.com/ProductDetail/AMD-Xilinx/A-U250-A64G-PQ-G?qs=unwgFEO1A6vi%2FTDpEsBygA%3D%3D), a [$10,000 evaluation kit](https://www.xilinx.com/products/boards-and-kits/vcu118.html) (although you can back-order the chip itself inside the evaluation board for a [cool $50,000](https://www.digikey.com/en/products/detail/amd/XCVU9P-L2FLGA2104E/7421960), which, uh 😵‍💫). But, [RHS Research](https://rhsresearch.com/pages/about-us) to the rescue! Their Nitefury II is a M.2 form-factor accelerator for $150, which is an awful lot more "in budget", even if it only seems to be available via [Amazon](https://www.amazon.com/dp/B0B9FMBF6C). Also of interest: https://www.crowdsupply.com/rhs-research & https://github.com/RHSResearchLLC/NiteFury-and-LiteFury . +- programming software: seems like an awful lot of _that_ stack is vendor-specific, locked-down, slow, lacking in Linux support (but there's always WINE), and/or varies greatly in quality. +- so-called "IP" availability: there's a somewhat eclectic mix of "parts" available to "plug in" to a design. + - it sure would be nice to have an index where we could say, e.g. "ETH PHY 1000" and get a list of gigabit-capable "core"s (not the CPU kind of "core"). Maybe someone should make that (us?). + - To be useful, it'd probably also need to be further filterable by "integration details," maybe HDL? "inward" facing protocols (SPI; TileLink; that ARM interlink protocol thing; etc.)? + # compliance ## being compliant From d808a5a24d349125fdaedf300eccb7b3b680a12d Mon Sep 17 00:00:00 2001 From: sethp Date: Sun, 12 May 2024 12:19:05 -0700 Subject: [PATCH 3/6] Add expected project benefits/risks Attempts to answer the question "who is this for" & "why do they want it?" --- riscv-router.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/riscv-router.md b/riscv-router.md index 8c326dc..2cde672 100644 --- a/riscv-router.md +++ b/riscv-router.md @@ -28,14 +28,30 @@ And some broader tensions that have impacts here: - Modularity and reliability - Instrumentation and performance +## Benefits + +Networking hardware is a core primitive of making useful share-able services. Learning more about these pieces and how they fit together is attractive for our goals. It's also a natural starting point/lynchpin for "home lab" enthusiasts, since it's the intemediation between all the other devices. That's a low volume market, but one with higher quality feedback as people self-select into their interest. + +As a target, building a router equips us to practice skills like RTL modeling & FPGA development that seem broadly applicable. There's many potential side-avenues to explore & find an adjacent niche (e.g. FPGA development accelerator boards), with plenty of seemingly low-hanging fruit. It's also got a clear success criteria & natural way for us to self-validate: it solves a problem we actually have. + +Starting at the network edge keeps performance requirements relatively low: while it's not unreasonable for people to want 10gig, 25gig or even 100gig performance "inside" their network, almost no small networks have an internet connection faster than gigabit. + +## Risks + +The small-scale router space is somewhat crowded; most of the entrants are highly locked down, which we know doesn't provide either reliability or security at the limit. But the difference in quality between a device that experiences a (very loosely defined) "fault" every 10,000 hours vs every 1 million hours is _very_ hard to percieve at the point of sale. Instead, feature & price sensitivity probably dominate (in what rough proportion with reputation? extensibility?). In other words, it's possible I (Seth) am the only person in the world who cares about having a router that is as open as possible, modular & upgradable, and still affordable/low power. + +There's also a _long_ list of "things people expect a network to do," with very little in the way of coherent overlap (i.e. there's a very strange topology of "what is possible" vs. "what is easy"; many roads lead _near_ the same places, but not to them). It's also historically a domain occupied by a small number of experts, and held at "arm's length" as much as possible. This defensivenes is understandable given the abysmally poor quality of feedback when experimenting with the system, but it's also a collective property that may be hard to overcome in a product (i.e. every device on the network shares the same unhelpful "internet test" button that usually provides almost no value). + ### Comparison + + @@ -73,11 +89,11 @@ What is the minimum clockspeed? Per https://en.wikipedia.org/wiki/Gigabit_Ethern ## Questions + * Do we want to make a router or switch? * What is Router? * What is Switch? -* What do *we* want it to do? -* Who do we think this is for? Why do they want it? +* What do we want it to do? * Name? * Open Switch (lol, like a electronic switch?) * Closed Switch (lol, irony here since we're open?) From cad5709dec946654e1b1e917b2bb576d62cb6d96 Mon Sep 17 00:00:00 2001 From: sethp Date: Sun, 12 May 2024 19:24:42 -0700 Subject: [PATCH 4/6] Add notes about IGLOO2 Gigabit ethernet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Everything but the price looks really promising; and the cost is ≥50% the PHY chip, for which I did almost no research/cost comparison. --- riscv-router.md | 46 ++++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/riscv-router.md b/riscv-router.md index 2cde672..377ee40 100644 --- a/riscv-router.md +++ b/riscv-router.md @@ -42,35 +42,51 @@ The small-scale router space is somewhat crowded; most of the entrants are highl There's also a _long_ list of "things people expect a network to do," with very little in the way of coherent overlap (i.e. there's a very strange topology of "what is possible" vs. "what is easy"; many roads lead _near_ the same places, but not to them). It's also historically a domain occupied by a small number of experts, and held at "arm's length" as much as possible. This defensivenes is understandable given the abysmally poor quality of feedback when experimenting with the system, but it's also a collective property that may be hard to overcome in a product (i.e. every device on the network shares the same unhelpful "internet test" button that usually provides almost no value). - ### Comparison
Design Goal EdgeRouter X EdgeRouter X-SFP
Price$40$60$100
Power (max)5W5W
- + - + - - - - - - - - + + + + + + + + + + + - - - + + + + + + +
Design GoalIgloo2 FPGA + Marvell 88E1340S 88E1543 PHY chip EdgeRouter X EdgeRouter X-SFPHex S
Price$40$60$100
Power (max)5W5W
Power (avg)??<5W ?
ISARISC-VMIPS
Networking(2-4) 10/100/1000 RJ45 ports(3) 10/100/1000 RJ45 ports
(1) 10/100/1000 RJ45 port (Data/PoE input)
(1) 10/100/1000 RJ45 port (Data/PoE passthrough)		(5) 10/100/1000 RJ45 ports
(1) 100/1000 SFP port
PoE Output??N/APassive 12W (24V)
Software??Linux (Vyatta)
Other Hardware?? +
Price$40BOM Cost: $31
[~$14 (FPGA) + ~$17 (PHY)]		$60$100$80
Power (max)5W~1 W
[500mW (FPGA) + 280 mW (PHY)]		5W (no SFP)6 W (24 W with SFP attachment)
Power (avg)????5W ???
ISARISC-VNone (RISC-V available?)MIPS
Networking(2-4) 10/100/1000 RJ45 portsup to (4) 10/100/1000 RJ45 Ports
(not priced)		(3) 10/100/1000 RJ45 ports
(1) 10/100/1000 RJ45 port (Data/PoE input)
(1) 10/100/1000 RJ45 port (Data/PoE passthrough)		(5) 10/100/1000 RJ45 ports
(1) SFP port
PoE Output??N/APassive 12W (24V)Passive 28.5W (up to 57V)
Software??CoreTSE "soft IP" ($4,000 per "seat")
or, ??		Linux (Vyatta)Linux, lol ("RouterOS", which is "allegedly" violating the GPL)
Other Hardware????
ProcessorDual-core 880 MHz, MIPS1004Kc
Memory256 MB DDR3 RAM
Storage256 MB NAND
 + + + + + +
ProcessorDual-core 880 MHz, MT7621A
Memory256 MB DDR3 RAM
Storage16 MB Flash
ExpansionmicroSD + USB
+
Ambient operating temperatureAt least +10 to 40°C-10 to 45° C (14 to 113° F)
Ambient operating humidity??10 to 90% noncondensing
Certifications??CE, FCC, IC
Ambient operating temperatureAt least +10 to 40°C??-10 to 45° C (14 to 113° F)-40°C to 70°C
Ambient operating humidity????10 to 90% noncondensing??
Certifications????CE, FCC, ICCE, EAC, ROHS
Notes + The 500mW estimate for the FPGA is extremely conservative; more typical numbers would seem to be an order of magnitude less. More work required to get better resolution.

Volume discounts for the chips start around 60-70 units and bring down the BOM cost by about $2.50

Gigabit Loopback Demo: https://www.microchip.com/en-us/application-notes/dg0633		 +