Skip to content

Commit 32485f0

Browse files
Kevin Wilfongfacebook-github-bot
Kevin Wilfong
authored andcommitted
feat: Replace tz implementation from date with LLVM implementation (#12422)
Summary: Pull Request resolved: #12422 TODO Differential Revision: D69997637
1 parent 88a05e3 commit 32485f0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+8168
-7550
lines changed

velox/docs/develop/timestamp.rst

+18-18
Original file line numberDiff line numberDiff line change
@@ -73,9 +73,9 @@ used to efficiently represent timezones, preventing the use of inefficient
7373
timezone string names like ``America/Los_Angeles``. Considering there are about
7474
2k valid timezone definitions, 12 bits are enough to represent timezone IDs. 
7575

76-
Timezone IDs in Velox are based on the id map used by Presto and are
77-
`available here <https://github.com/prestodb/presto/blob/master/presto-common/src/main/resources/com/facebook/presto/common/type/zone-index.properties>`_.
78-
They are automatically generated using `this script <https://github.com/facebookincubator/velox/blob/main/velox/type/tz/gen_timezone_database.py>`_.
76+
Timezone IDs in Velox are based on the id map used by Presto and are
77+
`available here <https://github.com/prestodb/presto/blob/master/presto-common/src/main/resources/com/facebook/presto/common/type/zone-index.properties>`_.
78+
They are automatically generated using `this script <https://github.com/facebookincubator/velox/blob/main/velox/type/tz/gen_timezone_database.py>`_.
7979
While timezone IDs are an implementation detail and ideally should not leak
8080
outside of Velox execution, they are exposed if data containing
8181
TimestampWithTimezones are serialized, for example.
@@ -98,7 +98,7 @@ ID).
9898

9999
However, unpacking/converting a TimestampWithTimezone into an absolute time
100100
definition requires a
101-
`timezone conversion <https://github.com/facebookincubator/velox/blob/main/velox/functions/prestosql/DateTimeFunctions.h#L74-L84>`_.
101+
`timezone conversion <https://github.com/facebookincubator/velox/blob/main/velox/functions/prestosql/DateTimeFunctions.h#L74-L84>`_.
102102

103103
Conversions Across Timezones
104104
----------------------------
@@ -119,7 +119,7 @@ for Linux. 
119119
In Velox, Timezone conversions are done using std::chrono. Starting in C++20,
120120
std::chrono `supports conversion of timestamp across timezones <https://en.cppreference.com/w/cpp/chrono/time_zone>`_.
121121
To support older versions of the C++ standard, in Velox we vendor an
122-
implementation of this API at `velox/external/date/ <https://github.com/facebookincubator/velox/tree/main/velox/external/date>`_.
122+
implementation of this API at `velox/external/tzdb/ <https://github.com/facebookincubator/velox/tree/main/velox/external/tzdb>`_.
123123
This class handles timezone conversions by leveraging APIs provided by the
124124
operating system, based on the tzdata database installed locally. If systems
125125
happen to have inconsistent or older versions of the tzdata database, Velox’s
@@ -153,23 +153,23 @@ on the string on not:
153153
::
154154

155155
SELECT typeof(TIMESTAMP '1970-01-01 00:00:00'); -- timestamp
156-
SELECT typeof(TIMESTAMP '1970-01-01 00:00:00 UTC'); -- timestamp with time zone
156+
SELECT typeof(TIMESTAMP '1970-01-01 00:00:00 UTC'); -- timestamp with time zone
157157

158158
Converting a TimestampWithTimezone into a Timestamp works by dropping the
159159
timezone information and returning only the timestamp portion:
160160

161161
::
162162

163-
SELECT cast(TIMESTAMP '1970-01-01 00:00:00 UTC' as timestamp); -- 1970-01-01 00:00:00.000
164-
SELECT cast(TIMESTAMP '1970-01-01 00:00:00 America/New_York' as timestamp); -- 1970-01-01 00:00:00.000
163+
SELECT cast(TIMESTAMP '1970-01-01 00:00:00 UTC' as timestamp); -- 1970-01-01 00:00:00.000
164+
SELECT cast(TIMESTAMP '1970-01-01 00:00:00 America/New_York' as timestamp); -- 1970-01-01 00:00:00.000
165165

166166
To convert a Timestamp into a TimestampWithTimezone, one needs to specify a
167167
timezone. In Presto, the session timezone is used by default:
168168

169169
::
170170

171171
SELECT current_timezone(); -- America/Los_Angeles
172-
SELECT cast(TIMESTAMP '1970-01-01 00:00:00' as timestamp with time zone); -- 1970-01-01 00:00:00.000 America/Los_Angeles
172+
SELECT cast(TIMESTAMP '1970-01-01 00:00:00' as timestamp with time zone); -- 1970-01-01 00:00:00.000 America/Los_Angeles
173173

174174
Conversion across TimestampWithTimezone can be done using the AT TIME ZONE
175175
construct. 
@@ -180,15 +180,15 @@ the clock/calendar read at the target timezone (Los Angeles)?
180180

181181
::
182182

183-
SELECT TIMESTAMP '1970-01-01 00:00:00 UTC' AT TIME ZONE 'America/Los_Angeles'; -- 1969-12-31 16:00:00.000 America/Los_Angeles
184-
SELECT TIMESTAMP '1970-01-01 00:00:00 UTC' AT TIME ZONE 'UTC'; -- 1970-01-01 00:00:00.000 UTC
183+
SELECT TIMESTAMP '1970-01-01 00:00:00 UTC' AT TIME ZONE 'America/Los_Angeles'; -- 1969-12-31 16:00:00.000 America/Los_Angeles
184+
SELECT TIMESTAMP '1970-01-01 00:00:00 UTC' AT TIME ZONE 'UTC'; -- 1970-01-01 00:00:00.000 UTC
185185

186186
Strings can be converted into Timestamp and TimestampWithTimezone:
187187

188188
::
189189

190-
SELECT cast('1970-01-01 00:00:00' as timestamp); -- 1970-01-01 00:00:00.000
191-
SELECT cast('1970-01-01 00:00:00 America/Los_Angeles' as timestamp with time zone); -- 1970-01-01 00:00:00.000 America/Los_Angeles
190+
SELECT cast('1970-01-01 00:00:00' as timestamp); -- 1970-01-01 00:00:00.000
191+
SELECT cast('1970-01-01 00:00:00 America/Los_Angeles' as timestamp with time zone); -- 1970-01-01 00:00:00.000 America/Los_Angeles
192192

193193
One can also convert a TimestampWithTimezone into a unix epoch/time. The
194194
semantic of this operation is: at the absolute point in time described by the
@@ -197,16 +197,16 @@ that unix epoch is the number of seconds since ``1970-01-01 00:00:00`` in UTC:
197197

198198
::
199199

200-
SELECT to_unixtime(TIMESTAMP '1970-01-01 00:00:00 UTC'); -- 0.0
201-
SELECT to_unixtime(TIMESTAMP '1970-01-01 00:00:00 America/Los_Angeles'); -- 28800.0
200+
SELECT to_unixtime(TIMESTAMP '1970-01-01 00:00:00 UTC'); -- 0.0
201+
SELECT to_unixtime(TIMESTAMP '1970-01-01 00:00:00 America/Los_Angeles'); -- 28800.0
202202

203203
The opposite conversion can be achieved using ``from_unixtime()``. The function
204204
may take an optional second parameter to specify the timezone, having the same
205205
semantic as AT TIME ZONE described above:
206206

207207
::
208208

209-
SELECT from_unixtime(0); -- 1970-01-01 00:00:00.000
209+
SELECT from_unixtime(0); -- 1970-01-01 00:00:00.000
210210
SELECT from_unixtime(0, 'UTC'); -- 1970-01-01 00:00:00.000 UTC 
211211
SELECT from_unixtime(0, 'America/Los_Angeles'); -- 1969-12-31 16:00:00.000 America/Los_Angeles
212212

@@ -225,8 +225,8 @@ timestamps have a different semantic:
225225
::
226226

227227
SET SESSION legacy_timestamp = true;
228-
SELECT cast(TIMESTAMP '1970-01-01 00:00:00 UTC' as timestamp); -- 1969-12-31 16:00:00.000
229-
SELECT cast('1970-01-01 00:00:00 UTC' as timestamp); -- 1969-12-31 16:00:00.000
228+
SELECT cast(TIMESTAMP '1970-01-01 00:00:00 UTC' as timestamp); -- 1969-12-31 16:00:00.000
229+
SELECT cast('1970-01-01 00:00:00 UTC' as timestamp); -- 1969-12-31 16:00:00.000
230230

231231
To support the two timestamp semantics, the
232232
``core::QueryConfig::kAdjustTimestampToTimezone`` query flag was added to Velox.

velox/dwio/parquet/tests/reader/ParquetTableScanTest.cpp

-1
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@
2525
#include "velox/exec/tests/utils/HiveConnectorTestBase.h" // @manual
2626
#include "velox/exec/tests/utils/PlanBuilder.h"
2727
#include "velox/exec/tests/utils/TempDirectoryPath.h"
28-
#include "velox/external/date/tz.h"
2928
#include "velox/type/tests/SubfieldFiltersBuilder.h"
3029
#include "velox/type/tz/TimeZoneMap.h"
3130

velox/expression/CastExpr-inl.h

-1
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919
#include "velox/common/base/Exceptions.h"
2020
#include "velox/core/CoreTypeSystem.h"
2121
#include "velox/expression/StringWriter.h"
22-
#include "velox/external/date/tz.h"
2322
#include "velox/type/Type.h"
2423
#include "velox/vector/SelectivityVector.h"
2524

velox/external/date/CMakeLists.txt

+1-3
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,5 @@
1010
# See the License for the specific language governing permissions and
1111
# limitations under the License.
1212

13-
14-
velox_add_library(velox_external_date tz.cpp)
13+
add_library(velox_external_date INTERFACE)
1514
velox_include_directories(velox_external_date SYSTEM PUBLIC velox/external)
16-
velox_compile_definitions(velox_external_date PRIVATE USE_OS_TZDB)

velox/external/date/ios.h

-56
This file was deleted.

velox/external/date/patches/0002-add_namespacing-pr-8332.patch

-100
Original file line numberDiff line numberDiff line change
@@ -23,106 +23,6 @@ diff --git a/fbcode/velox/external/date/date.h b/fbcode/velox/external/date/date
2323

2424
#ifdef _MSC_VER
2525
# pragma warning(pop)
26-
diff --git a/fbcode/velox/external/date/ios.h b/fbcode/velox/external/date/ios.h
27-
--- a/fbcode/velox/external/date/ios.h
28-
+++ b/fbcode/velox/external/date/ios.h
29-
@@ -32,16 +32,22 @@
30-
# if TARGET_OS_IPHONE
31-
# include <string>
32-
33-
+ namespace facebook
34-
+ {
35-
+ namespace velox
36-
+ {
37-
namespace date
38-
{
39-
namespace iOSUtils
40-
{
41-
-
42-
+
43-
std::string get_tzdata_path();
44-
std::string get_current_timezone();
45-
-
46-
+
47-
} // namespace iOSUtils
48-
} // namespace date
49-
+ } // namespace velox
50-
+ } // namespace facebook
51-
52-
# endif // TARGET_OS_IPHONE
53-
#else // !__APPLE__
54-
diff --git a/fbcode/velox/external/date/tz.cpp b/fbcode/velox/external/date/tz.cpp
55-
--- a/fbcode/velox/external/date/tz.cpp
56-
+++ b/fbcode/velox/external/date/tz.cpp
57-
@@ -268,6 +268,10 @@
58-
59-
#endif // !USE_OS_TZDB
60-
61-
+namespace facebook
62-
+{
63-
+namespace velox
64-
+{
65-
namespace date
66-
{
67-
// +---------------------+
68-
@@ -3860,6 +3864,8 @@
69-
}
70-
71-
} // namespace date
72-
+} // namespace velox
73-
+} // namespace facebook
74-
75-
#if defined(__GNUC__) && __GNUC__ < 5
76-
# pragma GCC diagnostic pop
77-
diff --git a/fbcode/velox/external/date/tz.h b/fbcode/velox/external/date/tz.h
78-
--- a/fbcode/velox/external/date/tz.h
79-
+++ b/fbcode/velox/external/date/tz.h
80-
@@ -143,6 +143,10 @@
81-
# endif
82-
#endif
83-
84-
+namespace facebook
85-
+{
86-
+namespace velox
87-
+{
88-
namespace date
89-
{
90-
91-
@@ -2790,5 +2794,7 @@
92-
#endif // !MISSING_LEAP_SECONDS
93-
94-
} // namespace date
95-
+} // namespace velox
96-
+} // namespace facebook
97-
98-
#endif // TZ_H
99-
diff --git a/fbcode/velox/external/date/tz_private.h b/fbcode/velox/external/date/tz_private.h
100-
--- a/fbcode/velox/external/date/tz_private.h
101-
+++ b/fbcode/velox/external/date/tz_private.h
102-
@@ -34,6 +34,12 @@
103-
#include <vector>
104-
#endif
105-
106-
+namespace facebook
107-
+{
108-
+
109-
+namespace velox
110-
+{
111-
+
112-
namespace date
113-
{
114-
115-
@@ -309,6 +315,10 @@
116-
117-
} // namespace date
118-
119-
+} // namespace velox
120-
+
121-
+} // namespace facebook
122-
+
123-
#if defined(_MSC_VER) && (_MSC_VER < 1900)
124-
#include "tz.h"
125-
#endif
12626
diff --git a/fbcode/velox/external/date/iso_week.h b/fbcode/velox/external/date/iso_week.h
12727
--- a/fbcode/velox/external/date/iso_week.h
12828
+++ b/fbcode/velox/external/date/iso_week.h

velox/external/date/patches/0003-throw-on-unsupported-tz-conversion-pr-10097.patch

-72
This file was deleted.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
diff --git a/velox/external/date/date.h b/velox/external/date/date.h
2+
index ccbd4587f..ac6d636dd 100644
3+
--- a/velox/external/date/date.h
4+
+++ b/velox/external/date/date.h
5+
@@ -1,5 +1,5 @@
6+
-#ifndef DATE_H
7+
-#define DATE_H
8+
+#ifndef VELOX_DATE_H
9+
+#define VELOX_DATE_H
10+
11+
// The MIT License (MIT)
12+
//
13+
@@ -7949,4 +7949,4 @@ operator<<(std::basic_ostream<CharT, Traits>& os,
14+
# pragma GCC diagnostic pop
15+
#endif
16+
17+
-#endif // DATE_H
18+
+#endif // VELOX_DATE_H

0 commit comments

Comments
 (0)