Skip to content

Optimze copy tensor with padding #32461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+76 −8
Open

Optimze copy tensor with padding #32461

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/plugins/intel_gpu/include/intel_gpu/runtime/layout.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,10 @@ struct layout {
// element == { 0,0,0,0 } means first no-padding (i.e. data) element
size_t get_linear_offset(tensor element = tensor(0)) const;

// Get variables needed for computing linear offset for a tensor with padding
void get_linear_offset_params(tensor& start_points, tensor& end_points, int64_t* padded_sizes,
int64_t* axes_map, size_t& map_size);

/// @brief Get aligned linear size calculated as multiplication of all elements.
size_t get_linear_size() const;

Expand Down Expand Up @@ -462,6 +466,8 @@ struct layout {
}

private:
static void get_axes_map(cldnn::format& fmt, int64_t* axes_map, size_t& map_size);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#32371 (comment)
Also, as I mentioned in the previous comment, please move this function to format.hpp

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can move get_linear_offset_params() to layout.hpp. May I know the reason or advantage you see to keep in hpp?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format.hpp, instead of layout.hpp. This is more about format. layout is user of format.
you can see there similar handles in format.hpp.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something likne format::internal_to_external(idx). The function you added is similar to that.


/// The size of the @ref memory (excluding padding)
ov::PartialShape size;
};
Expand Down
29 changes: 21 additions & 8 deletions src/plugins/intel_gpu/src/plugin/common_utils.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,29 @@ void convert_and_copy_no_pad(const src_t* src, dst_t* dst, size_t size) {
dst[i] = static_cast<dst_t>(src[i]);
}

#define MAX_NUM_AXES 6

template <typename src_t, typename dst_t>
void convert_and_copy_padded_source(const src_t* src, dst_t* dst, cldnn::layout layout) {
cldnn::tensor size = layout.get_tensor();
for (int64_t b = 0; b < size.batch[0]; b++) {
for (int64_t f = 0; f < size.feature[0]; f++) {
for (int64_t w = 0; w < size.spatial[3]; w++) {
for (int64_t z = 0; z < size.spatial[2]; z++) {
for (int64_t y = 0; y < size.spatial[1]; y++) {
for (int64_t x = 0; x < size.spatial[0]; x++) {
*dst++ = static_cast<dst_t>(src[layout.get_linear_offset(cldnn::tensor(b, f, x, y, z, w))]);
cldnn::tensor axes_start_point, axes_end_point;
int64_t padded_sizes[MAX_NUM_AXES], axes_map[MAX_NUM_AXES];
size_t map_len = MAX_NUM_AXES;

layout.get_linear_offset_params(axes_start_point, axes_end_point, padded_sizes, axes_map, map_len);

for (int64_t b = axes_start_point.batch[0]; b < axes_end_point.batch[0]; b++) {
for (int64_t f = axes_start_point.feature[0]; f < axes_end_point.feature[0]; f++) {
for (int64_t w = axes_start_point.spatial[3]; w < axes_end_point.spatial[3]; w++) {
for (int64_t z = axes_start_point.spatial[2]; z < axes_end_point.spatial[2]; z++) {
for (int64_t y = axes_start_point.spatial[1]; y < axes_end_point.spatial[1]; y++) {
for (int64_t x = axes_start_point.spatial[0]; x < axes_end_point.spatial[0]; x++) {
int64_t element_sizes[MAX_NUM_AXES] = {b, f, x, y, z, w};
size_t offset = element_sizes[axes_map[0]];

for (size_t i = 1; i < map_len; i++)
offset = offset * padded_sizes[i] + element_sizes[axes_map[i]];

*dst++ = static_cast<dst_t>(src[offset]);
}
}
}
Expand Down
49 changes: 49 additions & 0 deletions src/plugins/intel_gpu/src/runtime/layout.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,55 @@ std::vector<tensor::value_type> layout::get_pitches() const {
return pitches;
}

void layout::get_axes_map(cldnn::format& fmt, int64_t* axes_map, size_t& map_size) {
const auto& output_order = fmt.order();
const auto& internal_order = fmt.internal_order();
std::vector<int64_t> sizes_map(output_order.size(), 0);

//output_order has more elements than allocated in axes_map
if (output_order.size() > map_size) {
OPENVINO_THROW("Layout dimension higher than expected" + std::to_string(output_order.size()));
}

map_size = output_order.size();

for (size_t i = 0; i < map_size; i++) {
auto c = output_order[i];
auto pos = internal_order.find(c);

if (pos == std::string::npos)
OPENVINO_THROW("Unknown coord type: " + std::to_string(c));

axes_map[i] = pos;
}
}

void layout::get_linear_offset_params(tensor& start_points, tensor& end_points, int64_t* padded_sizes,
int64_t* axes_map, size_t& map_size) {
auto default_fmt = format::get_default_format(format.dimension(), format::is_weights_format(format), format::is_grouped(format));

std::vector<tensor::value_type> lower_sizes, upper_sizes;
lower_sizes.assign(data_padding._lower_size.begin(), data_padding._lower_size.begin() + format.dimension());
upper_sizes.assign(data_padding._upper_size.begin(), data_padding._upper_size.begin() + format.dimension());
start_points = tensor(default_fmt, lower_sizes, 0);
const auto& u_padd = tensor(default_fmt, upper_sizes, 0);

auto t = get_tensor();
end_points = t + start_points;

std::replace(t.raw.begin(), t.raw.end(), 0, 1);

get_axes_map(format, axes_map, map_size);
const auto& p_sizes = (t + start_points + u_padd).sizes(format);

if (p_sizes.size() < map_size) {
OPENVINO_THROW("Unsupported padded layout dimension" + std::to_string(p_sizes.size()));
}

for (size_t i = 0; i < p_sizes.size(); i++) {
padded_sizes[i] = p_sizes[i];
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, do we need this function inside the layout.cpp?
Seems that we can calculate jut in the target copy function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is using memeber variables of layout class. For example, data_padding. Even get_tensor() call has dependency to the layout object.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can get them from layout. Just like some_layout().data_padding.
Using memer is not a strong reason of having a new API.


size_t layout::get_linear_offset(tensor element) const {
auto default_fmt = format::get_default_format(format.dimension(), format::is_weights_format(format), format::is_grouped(format));
Expand Down
Loading