Replies: 14 comments 14 replies
-
I'm gonna group all of these into the same question, if that's ok, because the first three read to me like "is a hot dog a sandwich?", "what is a hot dog?", "what is a sandwich?" 🙃 Who it's for: me! I want it to replace one or both of the devices I currently have with RJ-45 jacks on the back, and those are doing two things:
|
Beta Was this translation helpful? Give feedback.
-
|
Surely at least one of your devices has wifi tho, yeah? We're not trying to make wifi are we? |
Beta Was this translation helpful? Give feedback.
-
|
Neither has on-board WiFi: one of the devices connected to the ISP-facing box (also known as an "edge" device) is a mostly stand-alone WiFi access point. The interface between the two is a virtual Linux switch device called So:
A box that's pure ethernet/802.3 solves a problem I have. I'd also like to replace the WiFi/802.11 device, at some point, but that's not on the list here. |
Beta Was this translation helpful? Give feedback.
-
|
I'm also realizing that maybe we should spend an hour or two breaking down the problem space together? Or we can run it async; I could put a skeleton of a plan together and send that to you to look over? I'm just thinking something like:
But to the next level of detail. What do you think? What would be helpful? I've also got some wilder hairs: fiber for an optical run to the TV room; breaking down capabilities into "modules" somehow; the aforementioned WiFi access point. Those'd be nice to capture, but I don't want to put them in the same place so as not to confuse the "broadening" and "narrowing" questions. |
Beta Was this translation helpful? Give feedback.
-
Heh, in my head I've been just thinking of it as "riscv router", the "open hardware, observable, hackable [as in hacking-on, not hacking-in] network device". The pronunciation is uncomfortably close to "risk-y router", because I am Good at Marketing. NOT. I like the plant theme; I'm not a botanist, but I have spent some time on Wikipedia, and that lead me to https://en.wikipedia.org/wiki/Lignin —that could be a fun name? |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if this is the same as "what do we want it to do" or not, but some areas of interest that I'd like to learn more about:
|
Beta Was this translation helpful? Give feedback.
-
|
These are just some nebulous thoughts I've been kinda having about the Router project. In service of:
I think the DIY open router is a good project. I do have some lingering "fears" (not actual fears) about the project though. Given my feeling here, I am glad you're taking the lead on this, but I also feel like I'm having a hard time fully wrapping my head around this. I feel like I don't have a place to crystalize any ideas, thoughts, or execution to help make much progress. Part of that is just an overall unfamiliarity with networking I think. I feel like it's hard to google for info or prior work done in this area. Most DIY "open" routers are implemented using ready made software designed to run in linux like on a laptop or even an existing router. So our goals are pretty different than usual DIY router projects. This leaves me feeling like there's not much to grab onto. But imagining myself researching the various ethernet protocols/languages we might need to speak to be a router and then writing some code to that spec feels pretty doable, even I don't really know what I'm doing - that's at least someplace to start. But that sort of assumes you have a whole set of hardware already ready. And I know how to read datasheets and learn a new chip, so even thinking about how to jostle the wires in a PHY chip seems daunting but doable - as does figuring out how to put all this on a board: doable but daunting/hard. All this as is feels like quite a large project, and maybe that's because it feels like there are a lot of unknowns (or maybe things that I feel like I don't know?). So then the idea of building our own SoC on top of it all feel very daunting indeed! I'm not at all saying we shouldn't do it that way, but just that I feel sort of at a loss of how to navigate us through all this but I'm happy to be mostly guided by you! Maybe you should take me on a tour of the Chipyard stuff? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for sharing you fears! It's definitely a big hairy ball of mud (if you'd enjoy a nice blended metaphor in the morning), and that's both good and bad: there's a lot to grab onto, but it's kind of slippery. I've got a clearer picture in my head about some parts than others, and the thing I'm definitely needing help with is turning that pile o' half-baked ideas into more concrete goals for the both of us. I think we're both a little shy about that, and probably for similar reasons. I really liked your idea yesterday about grabbing the parts I'd identified in #10 and seeing what it would take to slot them onto a PCB; the PCB design is something I definitely feel a lot fuzzier about, so I'm very glad you're much more confident about it! It is something I'd like to improve at, so I look forward to you guiding me through those bits like you helped me with some of the soldering/assembly in March.
It's true; "install and configure linux to do routing" is a very different shape of problem than I'm imagining for us here. I haven't found much in the way of open projects that try to do what I'm thinking of, which means there's not a lot of references to provide. It's also not particularly well-formed in my head, either: a lot of it is more ✨ vibes ✨ than, like, a consistent technical vision. (In my experience, "consistent technical vision" is always a survivorship-biased byproduct of vibes, but let's not get into that right now). What I do have is what I've been putting into the doc: things that are close-but-not-quite, with the intention of showing how what I'm thinking contrasts with it. Like: we talked about how we could get the
A lot of networking is indeed playing around with implementing protocols like you're imagining; and I do think it's worth trying to tease that learning apart from the hardware. Getting our heads around "implement network spec" feels pretty urgent to me, and I got lucky with Can we tuntap it?—that's a way to experiment with the protocols & start feeling out how they interact with zero additional hardware or software required.
I hear ya: there's a tradeoff in here that I'm keen to explore around "(re)programmability." The two extremes that I see are:
I'm curious about the middle: a more targeted SoC (rv64ic?) running bare-metal programs; that lets us write some "normal" code for a lot of the networking, and experiment with HDLs where they make sense. Though, probably this is worthy of a whole discussion on its own, so I'll break it off here. |
Beta Was this translation helpful? Give feedback.
-
|
One potential design: Igloo2 (FPGA) <-> GPY111 (Ethernet PHY) The igloo2 is definitely intended to be the MAC (controller) half of the equation. The IGLOO2 chips are super cheap, but I don’t know if they’d run a full risc-v core (not sure how many “logic elements” that would take). Still, they’d totally work as an entry-level part for proving out the concept of writing the “IP” for talking to the PHY chip, some sort of "make the FPGA do RGMII/SGMII" task The Igloo2 is low power & low cost, but I haven't been able to find any good dev boards for it. There was one that supported a RISC-V core (the Future Technologies Creative Board), but that's been discontinued it seems (it's impossible to find). So a big chunk of this design is the PCB work, with two main questions for me:
|
Beta Was this translation helpful? Give feedback.
-
|
Another design: one of the Xilinx FPGAs + the GPY111 They're all pretty pricey (the older gen Spartans are the only ones within the cost envelope, and I don't know much about them), but supposedly the Xilinx ecosystem is good, and there's some directly relevant bits to our goals:
They've also got a major downside over the Microsemi part: their FPGA configuration is stored in SRAM, which is fundamentally harder to wrangle in a low-power, on-demand mode than the Flash-backed Igloo2. |
Beta Was this translation helpful? Give feedback.
-
|
Less a design and more another way to get started in the microsemi ecosystem would be one of these:
the polarfire chips are too expensive (~$70 each), but they've got 5x (hard?) linux-capable risc-v cores pre-baked into the SoC. Either of those seems like they'd be a good dev kit for getting started with, and they're only ~$150. Both come with a pre-baked RJ-45 & Ethernet PHY too. getting one of these and getting it to respond to ICMPv4/v6 ECHO (ping) should take somewhere between a day and a week (full linux vs. bare-metal program), and then can be a fairly robust platform for doing at least some of the software-side development (neither seems to have 2x ethernet, although the beagle fire claims to somehow?). it would also be a way to dip a toe into the FPGA side and work on the RGMII/SGMII bits we'd need if we can poke at the onboard PHY from the I/O pins (assuming those line up with the Igloo2's) |
Beta Was this translation helpful? Give feedback.
-
|
One more possibility worth talking about: @dougli1sqrd raised the idea of hooking up the Allwinner D1 chip to an Ethernet PHY as the hardware side of the router. From the specs I can find, the D1 ought to speak RGMII, which suggests we could connect it to the GPY111. There's even a I am excited about the FPGA/RTL side of the universe, so it feels a little like a loss to me to lop off that portion of the project. That said, it's a good idea; we'd still have plenty to dig into around:
And it's worth noting, besides the Allwinner SoC, I haven't found any other existing RISC-V chips that would speak {R,S,}GMII . Theoretically, any such chip would be a candidate for integrating with the GPY111; there's probably a lot of valuable learning to be extracted from a comparison between the D1 and another SoC (including, possibly, our own). |
Beta Was this translation helpful? Give feedback.
-
|
So one of things you (@sethp, there's no one else here) were interested was gestures vaguely Hardware Acceleration for ethernet packets, and that's mainly what we'd want the FPGA for: to make some custom "hardware" that would make ethernet go fast. But perhaps we could think about this as a process, and not a fork in the road. If we built a board that uses a decently powerful risc-v off the shelf chip along side a PHY chip and whatever else needed to make that into a router, I imagine it could become clear exactly what sorts of acceleration we might want and what the pain points even are for being a router. Maybe we would want to continue to use the D1 actually and also use an FPGA that would be some kind of acceleration interface between phy and the CPU? Not sure. Or could be we would see exactly where we would want the acceleration in the logical diagram and realize the best place would be right next to a SoC of our making on the FPGA and we ditch the D1. But the D1 controlling the PHY or some variation like that could be an alpha version - proof of concept. Where we can then have a better understanding of how to incorporate an FPGA? |
Beta Was this translation helpful? Give feedback.
-
|
yeah, I dig this: now we're able to rub some differential diagnosis on the discussion. So there's three paths we're considering: a. Jumping "straight" to Igloo2 (FPGA) + GPY111 (Ethernet PHY); I think we're roughly in consensus that this is Too Hard, but for different definitions of Hard—which is great!1 Whether or not we actually converge at the Igloo2 + GPY111 isn't really that important to me, but I note that the D1 and PolarFire chips are both expensive enough (in price and possibly power) that they'd move us out of the $40/~1-5W "bucket"2. So, comparing b & c, I'm leaning towards b: it feels a lot more "available" to me. It sounds like you're leaning the other way, though: would you say more? I'm still not sure if we're seeking subtly different goals, or that we've got roughly the same goal and feel affinity for different paths. Like, I suspect we've got different impressions about how much progress we're making here in the abstract space: I'm feeling really good about it, but perhaps you're feeling less so? Footnotes
|
Beta Was this translation helpful? Give feedback.
-
Eric, there's you, me, and (hushed tones) posterity. |
Beta Was this translation helpful? Give feedback.
-
|
Aw cripes: I wrote the above operating on the assumption that the GPY111 would support four ethernet ports. It will not! The "4 transceivers" that I was going off of on mouser refer in this case (and, only this case) to the four twisted pairs that make up a single Ethernet connection: The GPY111 datasheet is still incredible, and contains lots of useful details for cross-referencing, but at $4 a part it's too expensive to support many more than 2-3 ports. It might still be good for a first-run prototype (especially a single-port "pingd" model), though. Some alternatives:
I haven't updated the comparison table yet to reflect this discovery, or to include either new option. That's important, since it'll probably have a big effect on the power envelope of the device. |
Beta Was this translation helpful? Give feedback.
-
How do router work maybe?so my understanding maybe is this:
The actual thing with the MAC address would be some combination of our FPGA+RISC-V+etc chip, and it's connected to the GSW141 via the SGMII (say) port. So to the GSW141 it's just another port to read/write ethernet frames to. It forwards the outside ethernet frame containing the IP packet from outside to the MAC (with our mac address). Then the MAC inspects the inner IP source/dest and constructs an ethernet frame to go to the right connection (computer actually attached via cat-5) which the GSW141 forwards to that computer |
Beta Was this translation helpful? Give feedback.
-
The magic of networking is that if we do it right, the device ought to be able to support arbitrary network graphs—including those with loops, though for now I'm only thinking about trees (i.e. no loops, at most one path between two nodes). I believe roughly what you've described should work for all of them, with one caveat below. Topologically, I think it makes sense to describe a few different stages:
Let's assume an ahead-of-time static IP config for both nodes (which we can sub out for DHCPv4 and [SLAAC/NDP or DHCPv6] later). The first one is nice and simple; then L1 sends an ARP/NDP request for L2's address, which is has a broadcast MAC address, so it hits the GSW chip which broadcasts it to L2, which says "hey that's me!" and then L1 knows what interface & MAC address to use for the ICMP echo request, and pings start flowing. The second I'm fuzzier on; I think it works just fine with no changes for L3->L1, since the ARP/NDP request that gets broadcast at SW2 hits SW1, which when broadcast again hits L1. I think L1 just responds to L3's MAC address, which works just in case SW1 and SW2 both "learned" the address as it went past. The third is where we need to get a little more involved, since I can almost guarantee that the OTP box (my ISP's device that corresponds to |
Beta Was this translation helpful? Give feedback.
-
|
A note about enclosures and ESD and RF nosie/radiation: This says that for a metal enclosure, we want the any "shielded" components to have their shields connected to a Net that connects to the enclosure via screws/standoffs that the circuit board sits on. Additionally, Signal Ground, our normal electric |
Beta Was this translation helpful? Give feedback.
-
|
Oh, that's very interesting: so GND and the shield are shorted together, and ought to be so through a very low impedance path (for both AC and DC), but outside of the circuit board. I'm not gonna lie, I'm struggling to build an effective mental model around the, what, field equations(?) going on there. Does that also stop noise propagation from inside out as well as outside in? |
Beta Was this translation helpful? Give feedback.
-
|
Haha yeah I think the goal is to prevent noise or interference getting on the board. So a metal enclosure will act kinda like a faraday cage for one thing and that's good. And everything would work if the connection was on the board, but then you'd potentially have noise entering the circuit board: if the enclosure gets a static charge, or receives an RF noise etc, you'd like it to just disappear into ground immediately. But if that noise had to first travel thru the pcb you're now radiating noise in the PCB. And yeah I think it would help stop noise getting out too! According to the EE overflow answer it's how PC motherboards work! so that's neat! |
Beta Was this translation helpful? Give feedback.
-
|
That makes sense to me. Looks like this plays into the jack selection question too, since I see some of those parts have spring-loaded tabs on the shield & others don't: that'd also be a way to establish a chassis connection, right? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah the EE overflow answer said that it's best if the shield can also make physical contact with your metal chassis |
Beta Was this translation helpful? Give feedback.
-
Router topologiesSo we've been making a board that lets us look into using the GSW141 Switch chip, as a component of a larger future router board. However, in the course of designing this "breakout" board, questions about power supplies, manufacturing capabilities by PCB shops, and just generally unknown unknowns as far as noise, signal impedance, etc. So now we've discussed using a different part, or maybe copying a plan that's already outlined (like in the TI https://www.ti.com/lit/ug/tiduef0a/tiduef0a.pdf doc). ---
title: One PHY chip per Port, Switch Chip uses xGMII
---
flowchart LR
rj45[RJ45 x4] <-- MDI x4 --> phy[PHY x4] <-- xGMII x4 --> Switch <-- xGMII --> mac["MAC" / Router CPU]
---
title: One PHY chip with 4 Ports, Switch Chip uses xGMII
---
flowchart LR
rj45[RJ45 x4] <-- MDI x4 --> phy[PHY x1] <-- xGMII x4 --> Switch <-- xGMII --> mac["MAC" / Router CPU]
---
title: Switch also contains PHY (like GSW141)
---
flowchart LR
rj45[RJ45 x4] <-- MDI x4 --> switch[Switch PHY] <-- xGMII --> mac["MAC" / Router CPU]
---
title: Switching and Routing all done in "software"
---
flowchart LR
rj45[RJ45 x4] <-- MDI x4 --> phy[PHY x1/x4] <-- xGMII x4 --> mac["MAC" / Router CPU]
Keep in mind the "MAC" box, and "done in software" could mean some kind of FPGA acceleration (which perhaps does the Switching etc?). The purpose of these diagrams is to understand all possible ways data could go in and out of whatever ultimately ends up being our processing solution for routing. There are a variety of chips that each do these things with a variety of tradeoffs: Cost and PCB manufacturability is top of mind for me. I think the thing that seems simplest is something like the 3rd entry above (like we're going for with the GSW141) and likely could be cheapest, just given it's also the smallest number of ICs?
|
Beta Was this translation helpful? Give feedback.
-
Yeah, agreed: the GSW141 was initially attractive to me because it seemed like it would reduce what I assumed would be our biggest cost—the ASICs we picked, which were O($10), and the GSW141 was cheaper than 4x GPY111s. Now that it looks like those are maybe among the smaller costs (or, at least, roughly equivalent to the others), a 50% or even 60% win there doesn't add up to as much as I thought overall. One more possibility that might be worth thinking about to de-risk both cost and manufacturability: a mini-board with 2x integrated magjacks, 2x GPY111s, and whatever power's required for the GPY111. If we just wire the xGMII interfaces "together" (so it's just logically a, like, longer cable) we might be able to make a relatively small (read: cheap) board that targets just the power supply & cross-talk questions. How does that strike you? |
Beta Was this translation helpful? Give feedback.
-
|
So I do think the GSW is the cheapest Switch chip out there, and it eliminates the need for any PHY. I think I'd like to explore how we can really make the GSW work. As far as a smaller mini breakout breakout, I'm hypothetically down, I'm just not sure it'd buy us much info? The power requirements would be so different than the GSW that I'm not sure it would help us in that domain. As far as checking magnetics and all that that could be worthwhile, but personally I think that's one of the less risky parts to me? That seems relatively straightforward so I haven't been too concerned about it. But definitely could do! |
Beta Was this translation helpful? Give feedback.
-
|
We talked a little about this today, but just to summarize a little bit: I do think the GPY111 would be an excellent set of training wheels from a power & cross-talk perspective. It has a lot of flexibility for the former and provides excellent detail on how to drive its internal switching regulator (which is optional, so we could decide to practice with both supplies or not). As for cross-talk, I'm not sure yet what to think: there seems to be a lot of similar-yet-distinct advice on how to do it, and the 8/10b encoding can correct 1-bit errors which gives us a lot of margin—although if we rely on that in the normal case that does come at the cost of general reliability. I'm excited to get into the details of that one next; for now I'll leave it at that. The GSW part is also good for getting experience with (relatively) high-pin-count ICs, and it does do a whole useful thing on its own. (I mean, an ethernet "extender"/signal booster is theoretically useful, I guess, if someone really wanted to run a single cable longer than 100m for whatever reason?). And, despite the bandwidth limitations, there's lots of really good reasons to plan on using the GSW part—especially if just to get something working so we can iterate from there! So I don't have a strong opinion yet between the two directions, but I'm open to both. |
Beta Was this translation helpful? Give feedback.
-
So Things that are Not Action Items But Kind of Look Like them if you Squint:
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
@dougli1sqrd asked some great questions in 366e747 ; let's talk about them!
Beta Was this translation helpful? Give feedback.
All reactions