So you are saying memory allocated by CL_MEM_USE_HOST_PTR is not visible to CPU? 
I am not convinced, because it doesnt make sense Where is it going to copy the data? The area is in host (uses CL_MEM_ALLOC_HOST_PTR) memory, and I am mapping it, which should simply give me the host pointer, and simply whatever I write there goes directly to where it should be. Therefore unmapping shouldnt need to copy anything.
If you check 4.6.2.2 carefully, you can see that 64MB limit is under CL_MEM_USE_PERSISTENT_MEM_AMD section only and indeed I am not able to allocate huge memory areas using that option.
The section 4.5.1.2 might be referring to automatical pinning of data under 32MB. However it is not very clear. Also the memory being pinned or not shouldnt have effect on zero-copy feature (there is no mention of that?
The example was purely for proving the point that zero-copy does not seem to be functioning properly. Of course if I code a real application, it will utilize whatever goes best for the application pattern.
yurtesen wrote:
I am not convinced, because it doesnt make sense
If you check 4.6.2.2 carefully, you can see that 64MB limit is under CL_MEM_USE_PERSISTENT_MEM_AMD section only and indeed I am not able to allocate huge memory areas using that option.
yeah that's what i said.
The section 4.5.1.2 might be referring to automatical pinning of data under 32MB. However it is not very clear. Also the memory being pinned or not shouldnt have effect on zero-copy feature (there is no mention of that?
The example was purely for proving the point that zero-copy does not seem to be functioning properly. Of course if I code a real application, it will utilize whatever goes best for the application pattern.
Clearly you're not getting zero copy behaviour. The guide just says it 'may' use it, not that it 'will always' use it. I've given a few reasons why it wouldn't be able to.
Hi yurtsen,
1. I would recommend you to use a clFinish() call before starting to measure the time of data transfer. A lot of other set-up time might also be included in your 1.3s otherwise. Or use events for enqueueWriteBuffers to get precision timing.
2. I don't think you are using the recommended path as suggested in Programming guide( unless it has changes recently 
) , You appear to be having the requirement of writing inside openCL buffers, whereas CL_MEM_ALLOC_HOST_PTR is a recommended flag when you need to read data from a openCL buffer(bacause of WC feature for such buffers).
3. In the situation above, i would recommend to use CL_MEM_USE_PERSISTENT_MEM_AMD, which allocates buffer in host accessible device memory. Writing to such a buffer should happen at peak interconnect speed.
gautam.himanshu wrote:
1. I would recommend you to use a clFinish() call before starting to measure the time of data transfer. A lot of other set-up time might also be included in your 1.3s otherwise. Or use events for enqueueWriteBuffers to get precision timing.
In this case, it is unnecessary since map operation is called with blocking flag true and there are no other OpenCL calls between the map until the timing starts. (Note that unnecessary clFinish() statements will make your code slower also.).
In either case, I put a clFinish() before the first gettimeofday call and the results are the same.
I found out that event times are unreliable and should not be used for measuring program performance. They seem to tell what you want to see only The actual runtime of the program code on the other hand is different. I made a more elaborate example to show it. I will put a 2nd example code which prints out all the timings separately. It appears the event timer does not include some of the time!
gautam.himanshu wrote:
2. I don't think you are using the recommended path as suggested in Programming guide( unless it has changes recently
I am not writing to the buffer from OpenCL kernel, I am writing from host code. The best performance is achieved if host code writes to pinned memory location in host. I know the part you mention in the manual, but it is not talking about about reading/writing to area using host code. This is the recommended usage for the example code.
gautam.himanshu wrote:
3. In the situation above, i would recommend to use CL_MEM_USE_PERSISTENT_MEM_AMD, which allocates buffer in host accessible device memory. Writing to such a buffer should happen at peak interconnect speed.
That would cause the memory location to reside on device memory. The peak interconnect speak in this case is the PCIe bandwidth, which is much less than using the CL_MEM_*_HOST_PTR which allows full memory bandwidth to write to the location by CPU.
See:
Page 4-14 Table 4.1
You can see that CPU accessing the device memory is extremely slow. In the code I used, only CPU accesses the memory location. Do you see the problem? (and that table is made based on PCIe3.0 systems)
$ g++ test.cpp -I./include -lOpenCL
$ time ./a.out
Tahiti
WALL time for map = 0.0 seconds
map #events: 3 time: 2e-09 seconds.
WALL time for for loops = 1.5 seconds
WALL time for unmap = 1.4 seconds
Unmap #events: 3 time: 2e-09 seconds.
real 0m3.765s
user 0m1.831s
sys 0m1.220s
$
As you can see, it appears the OpenCL code is trying to copy the memory to a different location at unmap (assumed from the time the operation takes). However, the manual clearly says this operation should be a zero-copy/no-op operation. Therefore there is a problem. If you think that there is something wrong in the code, please quote programming guide page number and title (so I can go and read it 
). In addition, unmap takes 0 seconds on windows system using the same code. Which means the zero-copy/no-op is working on windows only (although programing guide clearly mentions this working on Linux also!)
Also, you can see that the events are reporting 2e-09 seconds even though the operation took 1.4 seconds. Therefore the event timers are unreliable for measuring actual performance. They dont seem to include all the operations performed....
Updated code is below:
#include <iostream>
#include <fstream>
#include <iomanip>
#include <CL/cl.h>
#include <CL/cl_ext.h>
#include <CL/cl.hpp>
#include <sys/time.h>
using namespace std;
// For calculating time difference
#define TVDIFF(tv1, tv2) ((tv2).tv_sec - (tv1).tv_sec \
+ ((tv2).tv_usec - (tv1).tv_usec) * 1E-6)
void cl_checkelapsedtime(std::vector<cl::Event> events, const char *msg, int precision) {
cl_ulong startTime,endTime;
// Calculate elapsed time
cl::Event::waitForEvents(events);
startTime = events[0].getProfilingInfo<CL_PROFILING_COMMAND_START>();
endTime = events[events.size()-1].getProfilingInfo<CL_PROFILING_COMMAND_END>();
cl_ulong kernelExecTimeNs = endTime-startTime;
std::cout << msg << " #events: " << events.size() <<
" time: " << std::setprecision (precision) << (double)kernelExecTimeNs/1000000000 << " seconds."
<< std::endl;
}
int main() {
struct timeval tv1, tv2;
int size=11000*11000;
int device_id=1;
cl_int err;
std::vector<cl::Platform> platforms;
cl::Platform::get(&platforms);
cl_context_properties properties[] = { CL_CONTEXT_PLATFORM, (cl_context_properties)platforms[0](), 0};
cl::Context context(CL_DEVICE_TYPE_GPU, properties);
std::vector<cl::Device> devices = context.getInfo<CL_CONTEXT_DEVICES>();
cout << devices[device_id].getInfo<CL_DEVICE_NAME>() << endl;
cl_command_queue_properties queue_prop = 0;
cl::CommandQueue clqueue(context, devices[device_id], queue_prop, &err);
float *p_X,*p_Y,*p_Z;
cl::Buffer cl_X = cl::Buffer(context, CL_MEM_ALLOC_HOST_PTR, sizeof(float) * size, NULL, &err);
cl::Buffer cl_Y = cl::Buffer(context, CL_MEM_ALLOC_HOST_PTR, sizeof(float) * size, NULL, &err);
cl::Buffer cl_Z = cl::Buffer(context, CL_MEM_ALLOC_HOST_PTR, sizeof(float) * size, NULL, &err);
cl::Event event;
std::vector<cl::Event> clevents;
clqueue.finish();
gettimeofday(&tv1, NULL); /* Start measuring time */
p_X = (float *) clqueue.enqueueMapBuffer(cl_X, CL_TRUE, CL_MAP_WRITE, 0, sizeof(float) * size, NULL, &event, &err);
clevents.push_back(event);
p_Y = (float *) clqueue.enqueueMapBuffer(cl_Y, CL_TRUE, CL_MAP_WRITE, 0, sizeof(float) * size, NULL, &event, &err);
clevents.push_back(event);
p_Z = (float *) clqueue.enqueueMapBuffer(cl_Z, CL_TRUE, CL_MAP_WRITE, 0, sizeof(float) * size, NULL, &event, &err);
clevents.push_back(event);
clqueue.finish();
gettimeofday(&tv2, NULL); /* Stop measuring time */
printf("WALL time for map = %6.1f seconds\n", TVDIFF(tv1,tv2) );
cl_checkelapsedtime(clevents,"map",4);
clevents.clear();
gettimeofday(&tv1, NULL); /* Start measuring time */
for(int i=0;i<size;i++)p_X = 1.0;
for(int i=0;i<size;i++)p_Y = 2.0;
for(int i=0;i<size;i++)p_Z = 3.0;
gettimeofday(&tv2, NULL); /* Stop measuring time */
printf("WALL time for for loops = %6.1f seconds\n", TVDIFF(tv1,tv2) );
gettimeofday(&tv1, NULL); /* Start measuring time */
clqueue.enqueueUnmapMemObject(cl_X, p_X, NULL, &event);
clevents.push_back(event);
clqueue.enqueueUnmapMemObject(cl_Y, p_Y, NULL, &event);
clevents.push_back(event);
clqueue.enqueueUnmapMemObject(cl_Z, p_Z, NULL, &event);
clevents.push_back(event);
clqueue.finish();
gettimeofday(&tv2, NULL); /* Stop measuring time */
printf("WALL time for unmap = %6.1f seconds\n", TVDIFF(tv1,tv2) );
cl_checkelapsedtime(clevents,"Unmap",4);
clevents.clear();
return 0;
}
It appears there is an undocumented size limit on zero copy buffers on Linux while using CL_MEM_ALLOC_HOST_PTR and if the limit is passed, the SDK silently reverts to a very slow method... 
(confirmed by amd)
