How can I get non-tokenized data

[ztw33]

Hi,
First appreciate your excellent work on ACL and sharing this dataset with us!
I have a question and look forword to your reply. In file CoDesc.json I see original code and preprocessed code. So how can I get non-tokenized code, that is, its comments have been removed, but haven't been tokenized, which is grammatically correct in java and can be parsed into AST successfully?
Hope for your reply and thanks again!

[shaoyangxu]

Hi, First appreciate your excellent work on ACL and sharing this dataset with us! I have a question and look forword to your reply. In file CoDesc.json I see original code and preprocessed code. So how can I get non-tokenized code, that is, its comments have been removed, but haven't been tokenized, which is grammatically correct in java and can be parsed into AST successfully? Hope for your reply and thanks again!

Hello, can i ask you how to download the data?

I just ran the code from https://github.com/code-desc/CoDesc/blob/master/Setup/download_data.py
which is
from mega import Mega
mega = Mega()
mega.download_url('https://mega.nz/file/x5BQGDCY#LwmKDu5eYNTdG85xrW85jh3gcJvcsBpKwY9ufTFM1vs')

and i encountered below error:
Traceback (most recent call last):
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
raise err
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
conn = self._new_conn()
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 640, in download_url
return self._download_file(
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 658, in _download_file
file_data = self._api_request({
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 293, in wrapped_f
return self.call(f, *args, **kw)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 359, in call
do = self.iter(retry_state=retry_state)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 320, in iter
return fut.result()
File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 438, in result
return self.__get_result()
File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
raise self._exception
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 362, in call
result = fn(*args, **kwargs)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 166, in _api_request
response = requests.post(
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 117, in post
return request('post', url, data=data, json=json, **kwargs)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

[ztw33]

Hi, First appreciate your excellent work on ACL and sharing this dataset with us! I have a question and look forword to your reply. In file CoDesc.json I see original code and preprocessed code. So how can I get non-tokenized code, that is, its comments have been removed, but haven't been tokenized, which is grammatically correct in java and can be parsed into AST successfully? Hope for your reply and thanks again!

Hello, can i ask you how to download the data?

I just ran the code from https://github.com/code-desc/CoDesc/blob/master/Setup/download_data.py which is from mega import Mega mega = Mega() mega.download_url('https://mega.nz/file/x5BQGDCY#LwmKDu5eYNTdG85xrW85jh3gcJvcsBpKwY9ufTFM1vs')

and i encountered below error: Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn conn.connect() File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect conn = self._new_conn() File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 640, in download_url return self._download_file( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 658, in _download_file file_data = self._api_request({ File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 293, in wrapped_f return self.call(f, *args, **kw) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 359, in call do = self.iter(retry_state=retry_state) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 320, in iter return fut.result() File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 362, in call result = fn(*args, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 166, in _api_request response = requests.post( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 117, in post return request('post', url, data=data, json=json, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

I just found that the data provided by the author on MEGA has become invalid.🥲 I have downloaded the data before on my server. If you need, I could share it with you. How would you like to get it? It's large, about 6G.

[shaoyangxu]

Hi, First appreciate your excellent work on ACL and sharing this dataset with us! I have a question and look forword to your reply. In file CoDesc.json I see original code and preprocessed code. So how can I get non-tokenized code, that is, its comments have been removed, but haven't been tokenized, which is grammatically correct in java and can be parsed into AST successfully? Hope for your reply and thanks again!

Hello, can i ask you how to download the data?
I just ran the code from https://github.com/code-desc/CoDesc/blob/master/Setup/download_data.py which is from mega import Mega mega = Mega() mega.download_url('https://mega.nz/file/x5BQGDCY#LwmKDu5eYNTdG85xrW85jh3gcJvcsBpKwY9ufTFM1vs')
and i encountered below error: Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn conn.connect() File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect conn = self._new_conn() File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "", line 1, in File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 640, in download_url return self._download_file( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 658, in _download_file file_data = self._api_request({ File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 293, in wrapped_f return self.call(f, *args, **kw) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 359, in call do = self.iter(retry_state=retry_state) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 320, in iter return fut.result() File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 362, in call result = fn(*args, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 166, in _api_request response = requests.post( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 117, in post return request('post', url, data=data, json=json, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

I just found that the data provided by the author on MEGA has become invalid.🥲 I have downloaded the data before on my server. If you need, I could share it with you. How would you like to get it? It's large, about 6G.

Hi, First appreciate your excellent work on ACL and sharing this dataset with us! I have a question and look forword to your reply. In file CoDesc.json I see original code and preprocessed code. So how can I get non-tokenized code, that is, its comments have been removed, but haven't been tokenized, which is grammatically correct in java and can be parsed into AST successfully? Hope for your reply and thanks again!

Hello, can i ask you how to download the data?
I just ran the code from https://github.com/code-desc/CoDesc/blob/master/Setup/download_data.py which is from mega import Mega mega = Mega() mega.download_url('https://mega.nz/file/x5BQGDCY#LwmKDu5eYNTdG85xrW85jh3gcJvcsBpKwY9ufTFM1vs')
and i encountered below error: Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request self._validate_conn(conn) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn conn.connect() File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect conn = self._new_conn() File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "", line 1, in File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 640, in download_url return self._download_file( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 658, in _download_file file_data = self._api_request({ File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 293, in wrapped_f return self.call(f, *args, **kw) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 359, in call do = self.iter(retry_state=retry_state) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 320, in iter return fut.result() File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 438, in result return self.__get_result() File "/data1/syxu/anaconda3/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result raise self._exception File "/data1/syxu/anaconda3/lib/python3.9/site-packages/tenacity/init.py", line 362, in call result = fn(*args, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/mega/mega.py", line 166, in _api_request response = requests.post( File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 117, in post return request('post', url, data=data, json=json, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/data1/syxu/anaconda3/lib/python3.9/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='g.api.mega.co.nz', port=443): Max retries exceeded with url: /cs?id=4126649106 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fdac6d5cb80>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

I just found that the data provided by the author on MEGA has become invalid.🥲 I have downloaded the data before on my server. If you need, I could share it with you. How would you like to get it? It's large, about 6G.

Appreciate it!

You just said the data is on your server now, so could you just send it to my server, using:
scp -P 9998 your_data_path syxu@211.81.55.158:/data1/syxu

Thanks again!

QQ:1335911001
wechat:xsy15851559522

[Masum06]

Hi,

Thank you for reporting the issue. I am one of the authors. This repository was created for anonymous review and is no longer active. You will find the latest code on the following repository.
https://github.com/csebuetnlp/CoDesc

We have changed the broken Mega links to Google drive links. Now you should be able to download the dataset.