Messages in this thread | ![/](/images/icornerl.gif) | | Subject | Re: [PATCH net-next v3 3/4] page_pool: introduce page_pool_alloc() API | From | Yunsheng Lin <> | Date | Sat, 24 Jun 2023 23:39:14 +0800 |
| |
On 2023/6/20 23:39, Alexander Duyck wrote: ...
> >> If I understand it correctly, most hw have a per-queue fixed buffer >> size, even the mlx5 one with per-desc buffer size support through >> mlx5_wqe_data_seg, the driver seems to use the 'per-queue fixed >> buffer size' model, I assume that using per-desc buffer size is just >> not worth the effort? > > The problem is the device really has two buffer sizes it is dealing > with. The wqe size, and the cqe size. What goes in as a 4K page can > come up as multiple frames depending on the packet sizes being > received.
Yes, I understand that the buffer associated with wqe must be large enough to hold the biggest packet, and sometimes hw may report that only a small portion of that buffer is used as indicated in cqe when a small packet is received. The problem is: how much buffer is associated with a wqe to allow subdividing within wqe? With biggest packet being 2K size, we need a buffer with 4K size to be associated with a wqe, right? Isn't it wasteful to do that? Not to mention true size exacerbating problem for small packet.
And it seems mlx5 is not using the page_pool_fragment_page() API as you expected. As my understanding, for a mpwqe, it have multi strides, a packet seems to be able to fit in a stride or multi strides within a mpwqe, and a stride seems to be corresponding to a frag, and there seems to be no subdividing within a stride, see mlx5e_handle_rx_cqe_mpwrq().
https://elixir.bootlin.com/linux/v6.4-rc6/source/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c#L2366
...
> >>> >>> What I was thinking of was the frag count. That is something the >>> driver should have the ability to manipulate, be it adding or removing >>> frags as it takes the section of memory it was given and it decides to >>> break it up further before handing it out in skb frames. >> >> As my understanding, there is no essential difference between frag >> count and frag offet if we want to do 'subdividing', just like we >> have frag_count for page pool and _refcount for page allocator, we >> may need a third one for this 'subdividing'. > > There is a huge difference, and may be part of the reason why you and > I have such a different understanding of this. > > The offset is just local to your fragmentation, whereas the count is > the global value for the page at which it can finally be freed back to > the pool. You could have multiple threads all working with different > offsets as long as they are all bounded within separate regions of the > page, however they must all agree on the frag count they are working > with since that is a property specific to the page. This is why > frag_count must be atomic whereas we keep frag_offset as a local > variable. > > No additional counts needed. We never added another _refcount when we > were doing splitting in the drivers, and we wouldn't need to in order > to do splitting with page_pool pages. We would just have to start with > a frag count of 1.
In that case, we can not do something like below as _refcount if we have the same frag count for page pool and driver, right?
https://elixir.bootlin.com/linux/v6.4-rc6/source/drivers/net/ethernet/intel/iavf/iavf_txrx.c#L1220
| ![\](/images/icornerr.gif) |