Messages in this thread | | | Date | Fri, 10 Apr 2015 13:17:23 -0600 | From | Jason Gunthorpe <> | Subject | Re: [PATCH v2 01/17] IB/Verbs: Implement new callback query_transport() for each HW |
| |
On Fri, Apr 10, 2015 at 02:24:26PM -0400, Doug Ledford wrote:
> IPoIB is more than just an ULP. It's a spec. And it's very IB > specific. It will only work with OPA because OPA is imitating IB. > To run it on another fabric, you would need more than just to make > it work. If the new fabric doesn't have a broadcast group, or has > multicast registration like IB does, you need the equivalent of > IBTA, whatever that may be for this new fabric, buy in on the > pre-defined multicast groups and you might need firmware support in > the switches.
It feels like the 'cap_ib_addressing' or whatever we call it captures this very well. The IPoIB RFC is very much concerned with GID's and MGID's and broadly requires the IBA addressing scheme. cap_ib_addressing asserts the port uses that scheme.
We wouldn't accept patches to IPoIB to add a new addressing scheme without seeing proper diligence to the standards work.
Looking away from the stadards, using cap_XX seems very sane: We are building a well defined system of invarients, You can't call into the sa functions if cap_sa is not set, you can't call into the mcast functions if cap_mcast is not set, you can't form a AH from IB GIDs/MGID/LID without cap_ib_addressing.
I makes so much sense for the ULP to directly require the needed cap's for the kernel APIs it intends to call, or not use the RDMA port at all.
> > We can see how this might work in future, lets say OPAv2 *requires* the > > 32 bit LID, for that case cap_ib_address = 0 cap_opa_address = 1. If > > we don't update IPoIB and it uses the tests from above then it > > immediately, and correctly, stops running on those OPAv2 devices. > > > > Once patched to support cap_op_address then it will begin working > > again. That seems very sane.. > > It is very sane from an implementation standpoint, but from the larger > interoperability standpoint, you need that spec to be extended to the > new fabric simultaneously.
I liked the OPAv2 hypothetical because it doesn't actually touch the IPoIB spec. IPoIB spec has little to say about LIDs or LRHs it works entirely at the GID/MGID/GRH level.
Jason
| |