forked from apache/tez
-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathCHANGES.txt
2902 lines (2798 loc) · 206 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Apache Tez Change Log
=====================
Release 0.10.3: 2024-01-31
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-4526: Avoid calling LocationProvider#getPreferredLocations multiple times while generating grouped splits (#323)
TEZ-4039: Addendum: Tez should inject dag id, query id into MDC. (#325) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4520: Enable Parallel Compilation for TEZ (#315) (JiaLiangC reviewed by Laszlo Bodor)
TEZ-4494: Addressing CVE-2022-41881 - Netty vulnerability (#287) ( Sercan Tekin reviewed by Laszlo Bodor)
TEZ-4525: Remove broken links from site (#322)
TEZ-4524: Upgrade commons-codec to 1.15 (#321) (Bilwa S T reviewed by Laszlo Bodor)
TEZ-4522: Use OpenFile where FileStatus is available. (#318) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4523: Findbugs build is broken tez-tfile-parser. (#319) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4521: Partition stats should be always uncompressed size (#317) (okumin reviewed by Laszlo Bodor)
TEZ-3821: Ability to fail fast tasks that write too much to local disk. (#314) (Ayush Saxena reviewed by Rajesh Balamohan)
TEZ-4110: Make Tez fail fast when DFS quota is exceeded. (#313)
TEZ-4516: Fix log lines with incorrect placeholders. (#310) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4517: Upgrade commons-collection4 to 4.4. (#311) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4514: Reduce Some FileSystem Calls. (#309) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4506: Report the node of a task attempt failure better. (#307) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-2119: Counter for launched containers (#301) (Laszlo Bodor reviewed by Jonathan Eagles and Ayush Saxena)
TEZ-1037: Replace multiple members in TaskAttemptImpl for container related stuff with a single reference to the container. (#305) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4504: Upgrade Guava to 32.0.1 due to CVE-2023-2976. (#303) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4344: Collect jstack periodically from all containers. (#299) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4493: Upgrade Hadoop to 3.3.6. (#285) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4432: Remove useless maybeAddDefaultLoggingJavaOpts from TezClientUtils. (#297) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4472: Use unique DAG names in tests. (#293) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4473: Remove pointless MDC_LOGGING message from logs if there is no custom MDC key. (#294) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4250: Optimise TaskImpl::getCounters. (#295) (Ayush Saxena reviewed by Rajesh Balamohan)
TEZ-4478: Refactor dependency versions into properties in pom. (#291) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4495: mvnsite fails for some modules - No public or protected classes found to document (#288)
TEZ-4496: Tez precommit fails while building yetus image - no alternatives for some executables (#289)
TEZ-4470: typo fixes (#264) (Michal Lorek reviewed by Laszlo Bodor)
TEZ-4467: Migrate from Travis CI to GitHub Actions - pt2: remove .travis.yml (Laszlo Bodor reviewed by Ayush Saxena)
TEZ-4492: Update Bowerrc to use bower.herokuapp mirror to avoid Bower Registry CERT_EXPIRE issue (BOWER-2608) (#284) (Anmol Sundaram reviewed by Laszlo Bodor)
TEZ-4488: TaskSchedulerManager might not be initialized when the first DAG comes (#280) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4487: Add class name profiling option in ProfileServlet (#281) (Dmitriy Fingerman reviewed by Laszlo Bodor)
TEZ-4483: [CVE-2017-20165, CVE-2017-20165] Upgrade debug to 2.6.9 to fix the vulnerability (Aman Raj reviewed by Laszlo Bodor)
TEZ-4484: [CVE-2022-24999] Upgrade qs from 6.2.3 to 6.2.4 to fix the vulnerability (Aman Raj reviewed by Laszlo Bodor)
TEZ-4485: Upgrade jettison to 1.5.4 due to CVE-2023-1436 (#279) (Mayank Kunwar reviewed by Laszlo Bodor)
TEZ-4482: Upgrade JUnit to 4.13.2 due to medium CVEs (#276) (Mahesh Raju Somalaraju reviewed by Laszlo Bodor)
TEZ-4334: Fix deadlock in ShuffleScheduler between ShuffleScheduler.close() and the ShufflePenaltyReferee thread (#273) (Laszlo Bodor, Sungwoo Park, reviewed by Rajesh Balamohan)
TEZ-4460: Read timed out in shuffle handler - incorrect usage of EMPTY_LAST_CONTENT and channel write (#257) (Laszlo Bodor reviewed by Rajesh Balamohan, Syed Shameerur Rahman)
TEZ-4394: Netty4 ShuffleHandler: should use 1 boss thread (#256) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4469: Upgrade jettison to 1.5.3 to fix CVE-2022-45685, CVE-2022-45693 (#271) (Bilwa S T reviewed by Laszlo Bodor)
TEZ-4474: Added config to fail the DAG status when recovery data is missing (#266) (Mudit Sharma reviewed by Laszlo Bodor)
TEZ-4420: Upgrade Hadoop to 3.3.4 (#272) (Aman Raj reviewed by Laszlo Bodor)
TEZ-4467: Migrate from Travis CI to GitHub Actions (#270) - initial commit started from GitHub UI
TEZ-4475: VertexStatus is missing in TestLocalMode if DAG finishes too early - causing NPE in unit test (#267) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4397: Open Tez Input splits asynchronously (#263) (Syed Shameerur Rahman reviewed by Laszlo Bodor, original patch by Ramesh Kumar)
TEZ-4471: Precommit fails while building yetus image - upgrade to Ubuntu Jammy LTS (#265) (Laszlo Bodor reviewed by Rajesh Balamohan, Ayush Saxena)
TEZ-4462: update common-codec to 1.13 to fix vulnerability SNYK-JAVA-COMMONSCODEC-561518 (#258) (Dmitriy Fingerman reviewed by Rajesh Balamohan)
TEZ-4464: Update some links to https (#260) (Arnout Engelen reviewed by Laszlo Bodor)
TEZ-4458: Upgrade Bouncy Castle to 1.70 due to high CVEs (#253) (Mayank Kunwar reviewed by Laszlo Bodor)
TEZ-4456: Upgraded is-my-json-valid version to 2.20.3 (#250) (Aman Raj reviewed Laszlo Bodor)
TEZ-4396: Ensure utility classes have only private (default) constructors + several code refactors (#197) (Gergely Hanko reviewed by Laszlo Bodor)
TEZ-4455: Add LoggingHandler in ShuffleHandler pipeline for better debuggability. (#249) (Dmitriy Fingerman reviewed by Laszlo Bodor)
TEZ-4454: remove extra commas. (#246) (slfan1989 reviewed by Laszlo Bodor)
TEZ-4450: Shuffle data fetch fails when shuffle data is transferred via CompositeRoutedDataMovementEvent (#243) (Ganesha Shreedhara reviewed by Laszlo Bodor)
TEZ-4039: Tez should inject dag id, query id into MDC (#98) (Laszlo Bodor reviewed by Rajesh Balamohan, Prasanth Jayachandran)
TEZ-4453: Fix Typo of LocalClient. (#245) (slfan1989 reviewed by Laszlo Bodor)
TEZ-4448: Cannot submit Tez job when dag size is exceeds `ipc.maximum.data.length` and S3A is the filesystem (#241) (Karel Kolman reviewed by Laszlo Bodor)
TEZ-4449: Upgrade jettison to 1.5.1 to fix CVE-2022-40149. (#242) (fanshilun reviewed by Laszlo Bodor)
TEZ-4431: Apache Tez Release 0.10.2: update SNAPSHOT versions, DOAP file (#234) (Laszlo Bodor reviewed by Rajesh Balamohan, Ayush Saxena)
TEZ-4447: Collect VertexStatus properly when DAGClientServer is not used (local mode without network) (#239) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4441: TezAppMaster may stuck because of reportError skip send error event (#236) (zhengchenyu reviewed by Laszlo Bodor)
TEZ-4440: When tez app run in yarn fed cluster, may throw NPE (#235) (zhengchenyu reviewed by Laszlo Bodor)
TEZ-4435: use jackson v2 - jackson v1 is EOL and full of security issues (#231) (PJ Fanning reviewed by Laszlo Bodor, Ayush Saxena)
TEZ-4439: Update the protobuf documentation in TEZ codebase (#233) (Aman Raj reviewed by Laszlo Bodor)
TEZ-4363: Bump protobuf dependency to 3.x (#192) (Mark Bathori reviewed by Laszlo Bodor, Aman Raj)
Release 0.10.2: 2022-07-15
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-4430: Fix tez.task.launch.cmd-opts property not working (#228) (Ganesha Shreedhara reviewed by Laszlo Bodor)
TEZ-4427: [CVE-2019-10744] Upgrade lodash.merge version to 4.6.2 to fix vulnerability (#229) (Aman Raj reviewed by Nikhil Gupta, Laszlo Bodor)
TEZ-4426: [CVE-2018-1000620] Upgrade cryptiles from 2.0.5 to 4.1.2 (#227) (Aman Raj reviewed by Nikhil Gupta, Laszlo Bodor)
TEZ-4428: Use protoc-jar-maven-plugin to generate protobuf classes (#218) (Mark Bathori reviewed by Martin Tzvetanov Grigorov, Sylwester Lachiewicz, Laszlo Bodor)
TEZ-4429: Upgrade guava to 31.1 to fix CVE-2020-8908. (#223) (Ayush Saxena reviewed by Laszlo Bodor)
TEZ-4425: [WS-2020-0345] Upgrade jsonpointer version from 4.0.1 to 4.1.0 (#226) (Aman Raj reviewed by Laszlo Bodor)
TEZ-4424: [CVE-2021-3918] Upgrade json-schema from 0.2.3 to 0.4.0 to fix the vulnerability (#222) (Aman Raj reviewed by Laszlo Bodor)
TEZ-4423: [CVE-2021-44906] Upgrade minimist version from 0.0.8 to 1.2.6 to fix the vulnerability (#220)
TEZ-4422: [CVE-2021-43138] Upgrade async from 2.3.0 to 2.6.4 to fix the vulnerability. Also an upgrade of yarn version to 1.6.0 and frontend maven plugin to 1.8.0 was done. (#217) (Aman Raj reviewed by Laszlo Bodor)
TEZ-4038: Add a /prof profiler endpoint like HiveServer2 has (#212) (Zheenbek Akimzhanov reviewed by Laszlo Bodor)
TEZ-4383: upgrade to mockito 4.3.1 (#190) (Laszlo Attila Toth reviewed by Laszlo Bodor)
TEZ-4421: Missing sudo command in docker image (#214) (Mark Bathori reviewed by Laszlo Bodor)
TEZ-4409: Upgrade async-http-client to 2.12.3 to resolve CVE (#204) (D M Murali Krishna Reddy reviewed by Laszlo Bodor)
TEZ-4417: Missing wget command in docker image (#210) (Mark Bathori reviewed by Laszlo Bodor)
TEZ-4412: ensure mkDirForAM create directory with special permissions (#209) (Zhang Dongsheng reviewed by Laszlo Bodor)
TEZ-4347: Add some diagnostic endpoints to TezAM's WebUIService (#160) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4405: Replace log4j 1.x with reload4j (#200) (D M Murali Krishna Reddy reviewed by Laszlo Bodor)
TEZ-4403: Upgrade SLF4J Version To 1.7.36 (#198) (Syed Shameerur Rahman reviewed by Laszlo Bodor)
TEZ-4411: Update FileSaver dependency (#206) (Nikhil Gupta reviewed by Laszlo Bodor, Deependra Patel)
TEZ-4406: Use Path.getFilesystem api to get the correct FileSystem object for given staging path
TEZ-4398: Gitignore pyc files (#199) (Csaba Juhasz reviewed by Laszlo Bodor)
TEZ-4399: ShuffleHandler fails with SSLHandshakeException not found when SSL is enabled (#196) (Himanshu Mishra reviewed by Laszlo Bodor)
TEZ-3363: Delete intermediate data at the vertex level for Shuffle Handler (#60) (Syed Shameerur Rahman reviewed by Laszlo Bodor)
TEZ-4359: ShuffleHandler: Make sure of properly releasing netty reference counted objects (#194) (Mark Bathori reviewed by Laszlo Bodor)
TEZ-4388: Asynchttpclient can cause stuck TezChild processes (#189) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4300: Download protoc automatically compile/development time (#115)
TEZ-4389: TestSecureShuffle fails if localhost name is different than "localhost" (#191) (Mark Bathori reviewed by Laszlo Bodor)
TEZ-4384: Remove unused EnvironmentUpdateUtils remaining from TEZ-1837 (#186) (Laszlo Attila Toth reviewed by Laszlo Bodor)
TEZ-4379: Tez Hive Queries query page rows does not go over (#183) (Csaba Juhasz, Gergely Hanko reviewed by Laszlo Bodor)
TEZ-4227: Introduce convenient methods in TezID subclasses (#166) (Gergely Hanko reviewed by Laszlo Bodor)
TEZ-4381: Speed up TestSecureShuffle (#180) (Mark Bathori reviewed by Laszlo Bodor)
TEZ-4129: Delete intermediate attempt data for failed attempts for Shuffle Handler (#72) (Syed Shameerur Rahman reviewed by Laszlo Bodor)
TEZ-4350: Remove synchronized from DAGAppMaster.serviceInit (#162)
TEZ-4366: Disk utilization limit fix should be applied to all tez mini clusters (#173)
TEZ-4369: Upgrade netty to 4.1.72 due to CVE-2021-37136, CVE-2021-37137 (#175)
TEZ-4310: Avoid duplicated precommit comments (#176)
TEZ-4340: Show convenient input -> output vertex names in input messages (#170) (Csaba Juhasz reviewed by Laszlo Bodor)
TEZ-4376: Remove javax.security from the tez-auxservices shaded jar (#177) (Laszlo Bodor reviewed by Jonathan Eagles)
TEZ-4276: Clean Up Use of Preconditions (#96) (David Mollitor reviewed by Laszlo Bodor)
TEZ-4311: Bump hadoop dependency version to 3.3.x (#132) (Laszlo Bodor reviewed by Jonathan Eagles)
TEZ-4167: Speed up TestPipelinedSorter (#174) (Jonathan Eagles reviewed by Laszlo Bodor)
TEZ-4351: ShuffleHandler port should respect value in config (#163) (Laszlo Bodor reviewed by Jonathan Eagles)
TEZ-4364: TestFaultTolerance timeout on master (#171) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4354: Update netty to 4.1.61.Final (#164) (D M Murali Krishna Reddy reviewed by Laszlo Bodor)
TEZ-4349: DAGClient gets stuck with invalid cached DAGStatus (#161) (Ahmed Hussein reviewed by Laszlo Bodor, Jonathan Eagles)
TEZ-4270: HeartbeatHandlerBase Does Not Need Stop Condition (#93) (David Mollitor reviewed by Laszlo Bodor)
TEZ-4303: Exclude compile-time httpclient dependency from Tez (#119) (Laszlo Bodor reviewed by Jonathan Eagles)
TEZ-4294: Tez UI counters missing in settings view (#144) (Jonathan Eagles reviewed by Laszlo Bodor)
TEZ-4353: Update commons-io to 2.8.0 (#165) (D M Murali Krishna Reddy reviewed by Laszlo Bodor)
TEZ-4338: Tez should consider node information to realize OUTPUT_LOST as early as possible - upstream(mapper) problems (#152) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4339: Expose real-time memory consumption of AM and task containers via DagClient (#157) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4336: ShuffleScheduler should try to report the original exception (when shuffle becomes unhealthy) (#155) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4342: TestSecureShuffle is broken - No subject alternative names present (#158) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4180: Show convenient input -> output vertex names in output/sort messages (#154)
TEZ-4332: Change ASF logo on Tez webpage according to recent guidelines (#148)
TEZ-4329: Import external tez component em-table (#146)
TEZ-4328: Import external tez component em-helpers (#145)
TEZ-4323: Jetty jars were removed from dist package with TEZ-4114 (#140) (Takanobu Asanuma reviewed by Jonathan Turner Eagles)
TEZ-4293: Use Tez credentials for CI (#141)
TEZ-4321: Do Not Wrap Singleton Collections (#138)
TEZ-4231: Fix multiple history parser and event converter issues (#123) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4214: Upgrade RoaringBitmap version to 0.7.45 (#74)
TEZ-4125: Upgrade to Hadoop 3.2.x (#126)
TEZ-4314: Fix some plugin versions in site generation (#131)
TEZ-4267: Remove Superfluous Code from DAGAppMaster (#90)
TEZ-4313: Apache Tez Release 0.10.1 (#134)
TEZ-4308: Add Whitespace in ShuffleScheduler Error Message (#122)
TEZ-4275: Use Google Guava Intern Facility (#95)
TEZ-3918: Setting tez.task.log.level does not work
TEZ-4298: ShuffleHandler is not source compatible with Hadoop 3.3.0 (#129)
Release 0.10.1: 2021-07-01
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-4295: Could not decompress data. Buffer length is too small. (#130) (Laszlo Bodor reviewed by Ashutosh Chauhan)
TEZ-4309: TezUtils.addToConfFromByteString throws com.google.protobuf.CodedInputStream exception (Ramesh Kumar Thangarajan via László Bodor)
TEZ-4305: Check StreamCapabilities before using HFLUSH from ProtoMessageWriter (#120) (Kishen Das reviewed by Laszlo Bodor and Harish JP)
TEZ-4296: Use listStatusIterator instead of listStatus in DatePartitionedLogger (#124)
TEZ-4157: ShuffleHandler: upgrade to Netty4 and remove Netty3 dependency from tez (#118) (Laszlo Bodor reviewed by Ashutosh Chauhan, Jonathan Turner Eagles)
TEZ-4279: Add vertexId into vertex status for dag clients (#101) (Laszlo Bodor reviewed by Ashutosh Chauhan)
TEZ-4302: NullPointerException in CodecUtils with GzipCodec (#117) (Xi Chen reviewed by Laszlo Bodor)
TEZ-4299: Default java opts cause jdk11 to fail (#116) (Laszlo Bodor reviewed by Jonathan Turner Eagles)
TEZ-4291: Fancy precommit report on PR (#112)
TEZ-4289: Remove Dependency on commons-math (#110)
TEZ-4286: Consider sending fewer emails from github to dev@tez.apache.org (#107) (Laszlo Bodor reviewed by Jonathan Turner Eagles)
TEZ-4025: javadoc compilation is broken in jdk11 (review by Bodor Laszlo)
TEZ-4288: Upgrade SLF4J to 1.7.30 (#109)
TEZ-4241: UI: containerLogs testWrapper test fails (review by Bodor Laszlo)
TEZ-4259: Create Jenkinsfile for use with tez-multibranch (github yetus integration) (Jonathan Turner Eagles reviewed by Laszlo Bodor)
TEZ-4290: Dockerfile improvements (#111) (Laszlo Bodor reviewed by Jonathan Turner Eagles)
TEZ-3966: Fix config file mime type (#26)
TEZ-4278: Show PR link in Jira (#99)
TEZ-4282: Possible NPE in LocalClient after TEZ-4236 (#103) (Laszlo Bodor reviewed by Panagiotis Garefalakis, Ashutosh Chauhan)
TEZ-4281: dag_*_priority.dot files should go into a valid log directory (#102) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4273: Clear off staging files when TezYarnClient is unable to submit applications (Rajesh Balamohan via László Bodor) - addendum test fix
TEZ-4283: Docker fails to build on master - upgrade to nodejs10 (#104)
TEZ-4273: Clear off staging files when TezYarnClient is unable to submit applications (Rajesh Balamohan via László Bodor)
TEZ-4277: AsyncDispatcher can hang on serviceStop if the eventhandling thread is in BLOCKED state (#97) (Laszlo Bodor reviewed by Rajesh Balamohan)
TEZ-4266: Make Tez Inherit From Apache POM (#89)
TEZ-4236: DAGClientServer is not really needed to be started/used in local mode (László Bodor reviewed by Ashutosh Chauhan)
TEZ-4269: Re-Work Threadpool in DAGAppMaster (#92)
TEZ-4240: Remove SHA-256 from Tez (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-3985: Correctness: Throw a clear exception for DMEs sent during cleanup (Contributed by Jaume M, reviewed by Rajesh Balamohan, Ashutosh Chauhan)
TEZ-4268: Buffer File Read of TEZ AM Local Resources PB File (#91)
TEZ-4264: Make vertex id available from MROutputCommitter (Marton Bod via László Bodor)
TEZ-4265. Tez UI doesn't show Load Counters button if any of the Dags have them
TEZ-4095: Review of Debug Logging (#87)
TEZ-4261: Fix Javadoc warnings
TEZ-3706: add option to skip Tez UI build
TEZ-4258: Fix travisci build
TEZ-4253: Revert TEZ-4170 (Mustafa İman via Attila Magyar, Ashutosh Chauhan)
TEZ-4256: Reduce key comparisons in reducer side (Rajesh Balamohan via Gopal V)
TEZ-4244: Consider using RawLocalFileSystem in LocalDiskFetchedInput (Rajesh Balamohan reviewed by László Bodor)
TEZ-4251: Acquiring locks for getInputVertices and getOutputVertices is not consistent (Krisztian Kasa reviewed by Rajesh Balamohan)
TEZ-4248: MRReaderMapred should propagate properties based on config (Marton Bod via László Bodor)
TEZ-4237: Upgrade async-http-client-1.9.40 due to CVE-2017-14063 (László Bodor reviewed by Ashutosh Chauhan)
TEZ-4070: SSLFactory not closed in DAGClientTimelineImpl caused native memory issues (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4229: Improve TezLocalCacheManager to use configured root directory (László Bodor reviewed by Panagiotis Garefalakis, Ashutosh Chauhan)
TEZ-4238: Check null mrReader in MRInput.close (László Bodor reviewed by Hadoop QA, Jonathan Turner Eagles, Hadoop QA)
TEZ-4234: Compressor can cause IllegalArgumentException in Buffer.limit where limit exceeds capacity (László Bodor reviewed by Rajesh Balamohan, Jonathan Turner Eagles)
TEZ-4233: Map task should be blamed earlier for local fetch failures (Laszlo Bdoor reviewed by Rajesh Balamohan)
TEZ-4230: LocalContainerLauncher can kill task future too early, causing app hang (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4228: TezClassLoader should be used in TezChild and for Configuration objects
Release 0.10.0: 2020-10-15
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-4238: Check null mrReader in MRInput.close (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4234: Compressor can cause IllegalArgumentException in Buffer.limit where limit exceeds capacity (László Bodor reviewed by Rajesh Balamohan, Jonathan Turner Eagles)
TEZ-4230: LocalContainerLauncher can kill task future too early, causing app hang (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4228: TezClassLoader should be used in TezChild and for Configuration objects
TEZ-3645: Reuse SerializationFactory while sorting, merging, and writing IFiles (Jonathan Turner Eagles reviewed by Rajesh Balamohan, Laszlo Bodor)
TEZ-4175: Consider removing YarnConfiguration where it's possible (László Bodor reviewed by Rajesh Balamohan, Mustafa Iman, Ashutosh Chauhan)
TEZ-4224: Add Laszlo Bodor's public key to KEYS (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4213: Bound appContext executor capacity using a configurable property (Panagiotis Garefalakis reviewed by Ashutosh Chauhan, Mustafa Iman, Attila Magyar) - addendum checkstyle
TEZ-4216 : RLE check in MergeManager::finalMerge could be disabled (Rajesh Balamohan via Ashutosh Chauhan)
TEZ-4213: Bound appContext executor capacity using a configurable property (Panagiotis Garefalakis reviewed by Ashutosh Chauhan, Mustafa Iman, Attila Magyar)
TEZ-4207: Provide approximate number of input records to be processed in UnorderedKVInput (Rajesh Balamohan, reviewed by Ashutosh Chauhan)
TEZ-4223 - Adding new jars or resources after the first DAG runs does not work.
TEZ-4188. Link to NodeManager Logs of Home and DAG details doesn't consider yarnProtocol
TEZ-4208 : Pipelinesorter uses single SortSpan after spill (Rajesh Balamohan via Ashutosh Chauhan)
TEZ-4212. Fix build checkstyle configuration and suppressions dtd URLs (Jonathan Eagles reviewed by László Bodor)
TEZ-4172: Let tasks be killed after too many overall attempts (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4203. Findbugs: MergeThread.shuffleSchedulerThread; locked 80% of time
TEZ-4204: Data race in RootInputInitializerManager (Mustafa Iman via Ashutosh Chauhan)
TEZ-4206: TestSpeculation.testBasicSpeculationPerVertexConf is flaky (Mustafa Iman via Ashutosh Chauhan)
TEZ-4133. key class implements writableComparable and configurable use default configuration (wang qiang via jeagles)
TEZ-4199. MergeManager::finalMerge should make use of compression
TEZ-4201. findbugs-maven-plugin is not compatible with Maven 3.6.0+
TEZ-4200. Precommit docker image build fails
TEZ-4170 : RootInputInitializerManager could make use of ThreadPool from appContext ( Attila Magyar via Rajesh Balamohan)
TEZ-4185 : Tez may skip file permission update on intermediate output (Attila Magyar via Ashutosh Chauhan)
TEZ-4105: Tez job-analyzer tool to support proto logging history (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4135: Improve memory allocation when executing in-memory reads (László Bodor reviewed by Ashutosh Chauhan)
TEZ-4137 : Input/Output/Processor should merge payload to local conf (Mustafa Iman via Laszlo Bodor, Ashutosh Chauhan)
TEZ-4087 : Shuffle: Fix shuffle cleanup to prevent thread leaks (Rajesh Balamohan via Prasanth J, Ashutosh Chauhan)
TEZ-4179: [Kubernetes] Extend NodeId in tez to support unique worker identity (Prasanth Jayachandran, Attila Magyar, reviewed by Rajesh Balamohan)
TEZ-4186: Limits: Fix init order regression from TEZ-4155 (Gopal V, reviewed by Rajesh Balamohan)
TEZ-4171. DAGImp::getDAGStatus should try to report RUNNING state information correctly
TEZ-4182. Expose build user and java version in version-info.properties
TEZ-4174: [Kubernetes] Fetcher should connection failure on SocketException (Prasanth Jayachandran reviewed by Rajesh Balamohan)
TEZ-2672: Allow specifying a new payload for plugins when a new DAG starts (László Bodor reviewed by Rajesh Balamohan)
TEZ-4173: isSetParallelismCalled should be checked before skipping vertex reinit (Syed Shameerur Rahman via László Bodor)
TEZ-4165. Speed up TestShuffleScheduler#testNumParallelScheduledFetchers
TEZ-4164. Speed up TestFetcher
TEZ-4163. Speed up TestTaskReporter
TEZ-4162. Speed up TestInputReadyTracker
TEZ-4161. Speed up TestTezUtils
TEZ-4156: Fix Tez to reuse IPC connections (Rajesh Balamohan, reviewed by Siddharth Seth, László Bodor, Jonathan Turner Eagles)
TEZ-4158: Change to a maintained bouncy castle version (László Bodor reviewed by Ashutosh Chauhan, Jonathan Turner Eagles)
TEZ-4155: Remove sync bottleneck in counters (László Bodor reviewed by Rajesh Balamohan)
TEZ-4136: String representation for tez counters (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4151. Missing apache commons collections4 dependency on tez-mapreduce and tez-plugins/tez-history-parser (jeagles)
TEZ-4140: Tez DAG Recovery: Discrepancy In Scheduling Vertices During Vertex Recovery (Syed Shameerur Rahman via László Bodor)
TEZ-4147. Reduce NN calls in RecoveryService::handleRecoveryEvent
TEZ-4144: Checkstyle: '{' is followed by whitespace
TEZ-4143: Provide an option to disable DAG graph (.dot) generation for latency sensitive jobs (László Bodor reviewed by Rajesh Balamohan)
TEZ-4146: Register RUNNING state in DAG's state change callback (Rajesh Balamohan, reviewed by Gopal V)
TEZ-4145: Reduce lock contention in TezSpillRecord (László Bodor reviewed by Ashutosh Chauhan, Jonathan Turner Eagles, Rajesh Balamohan)
TEZ-4142. TezUtils.createConfFromByteString on Configuration larger than 32MB throws com.google.protobuf.CodedInputStream exception
TEZ-4134. Upgrade maven surefire plugin to 3.0.0-M4
TEZ-4131: Reduce apache commons collections direct dependencies
TEZ-4114: Remove direct jetty dependency from tez (László Bodor reviewed by Ashutosh Chauhan, Jonathan Turner Eagles)
TEZ-4097: Report localHostname in Fetcher and FetcherOrderedGrouped failure log messages (László Bodor reviewed by Ashutosh Chauhan)
TEZ-4127. TestMergeManager#testOnDiskMergerFilenames fails with dot directory name
TEZ-4099: Add details on whether SSL is enabled or not in HttpConnectionParams (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-3727: When using HDFS federation, token of tez.simple.history.logging.dir is not added, causing AM to fail (contributed by Xi Chen, reviewed by László Bodor, Jonathan Turner Eagles)
TEZ-3664. Flaky tests due to writing to /tmp directory
TEZ-4126: Shell scripts under tez-tools should be runnable (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4109: Improve TezCommonUtils.getCredentialsInfo and use it from more contexts (László Bodor reviewed by Ashutosh Chauhan)
TEZ-4100. Upgrade to Hadoop 3.1.3 and Guava 27
TEZ-4123. TestMRRJobsDAGApi flaky timeout - unhealthy node
TEZ-4124. GuavaShim: introduce an interoperability layer for different guava versions
TEZ-4122. TestMRRJobsDAGApi should set TezClassLoader
TEZ-4081. Container release idle timeout exception for equal min and max values
TEZ-4026. Fetch Download rate shows 0.0 MB per second if duration is 0 millis
Revert "TEZ-4082. Reduce excessive getFileLinkInfo calls in Tez"
TEZ-2229. bower ESUDO Cannot be run with sudo -- during build
TEZ-4106. Add Exponential Smooth RuntimeEstimator to the speculator
TEZ-4101. Eliminate some guava dependencies by Java8+ features - Preconditions
TEZ-4117: Fix minor issues in docs/pom.xml (László Bodor reviewed by Jonathan Turner Eagles)
TEZ-4102: Let session credentials be merged before merging am launch credentials (László Bodor reviewed by Ashutosh Chauhan)
TEZ-3391. Optimize single split MR split reader
TEZ-4113: TezUtils::createByteStringFromConf should use snappy instead of DeflaterOutputStream (rbalamohan, reviewed by Ashutosh Chauhan)
TEZ-4115: turn on data-via-events as default (Contributed by Richard Zhang, reviewed by rbalamohan)
TEZ-4098: tez-tools improvements: log-split, swimlane (László Bodor, reviewed by rbalamohan)
TEZ-4103. Progress in DAG, Vertex, and tasks is incorrect
TEZ-4108. NullPointerException during speculative execution race condition
TEZ-1869. Exclude tez-ui war / other dependencies from tez.tar.gz
TEZ-4107. PreCommit-TEZ-Build fails - Docker failed to build yetus/tez
TEZ-4067. Tez Speculation decision is calculated on each update by the dispatcher
TEZ-3860. JDK9: ReflectionUtils may not use URLClassLoader
TEZ-4083. Upgrade to latest 9.3.x Jetty version
TEZ-3992. Update commons-codec from 1.4 to 1.11
TEZ-4084. Tez local mode fails when distributed cache creates link with parent
TEZ-4085. Tez UI resources vendor.js and tez-ui.js not getting minified in tez releases (Himanshu Mishra via jeagles)
TEZ-4096: SSLFactory should pickup configs from incoming conf payload (rbalamohan, reviewed by gopalv)
TEZ-4091: UnorderedPartitionedKVWriter::readDataForDME should check if in-mem file is flushed or not (#53)
TEZ-4088: Create in-memory ifile writer for transferring smaller payloads (follow up of TEZ-4075)
TEZ-4075: Reimplement tez.runtime.transfer.data-via-events.enabled (#48) (Contributed by Richard Zhang)
TEZ-4086. Allow various examples to work when outputPath is on a FileSystem other than the default FileSystem. (#45)
TEZ-4082. Reduce excessive getFileLinkInfo calls in Tez
TEZ-4076. Add hadoop-cloud-storage jar to aws and azure mvn profiles (Jesus Camacho Rodriguez, reviewed by Gopal V)
TEZ-4068. Prevent new speculative attempt after task has issued canCommit to an attempt
TEZ-4066. Upgrade servlet-api from 2.5 to 3.1.0 (Jonathan Eagles via kshukla)
TEZ-1348. Allow Tez local mode to run against filesystems other than local FS. (Todd Lipcon via sseth)
TEZ-4062. Speculative attempt scheduling should be aborted when Task has completed
TEZ-4058. Changes for 0.9.2 release
TEZ-4057: Fix Unsorted broadcast shuffle umasks (Eric Wohlstadter, reviewed by Gopal V)
TEZ-4045. Task should be accessible from TaskAttempt
TEZ-4031. Support tez gitbox migration (Jonathan Eagles via kshukla)
TEZ-4052. Fit dot files ASF License issues - part 2 (Jonathan Eagles via kshukla)
TEZ-4044. Zookeeper: exclude jline from Zookeeper client from tez dist
TEZ-3995. Fix dot files produced by tests to prevent ASF license warnings in yetus (addendum)
TEZ-4048. Make proto history logger queue size configurable
TEZ-4047. Tez trademark in xml is causing xml parsing issue (Jonathan Eagles via kshukla)
TEZ-4050. maven site is failing due to missing configuration. (Jonathan Eagles via kshukla)
TEZ-4049. Fix findbugs issues in NotRunningJob (Jonathan Eagles via kshukla)
TEZ-4032. TEZ will throw Client cannot authenticate via:[TOKEN, KERBEROS] when used with HDFS federation(non viewfs, only hdfs schema used).
TEZ-4042. Speculative attempts should avoid running on the same node
TEZ-4035. Tez master breaks with YARN 3.2.0 ApplicationReport API change (jeagles)
TEZ-3952. Allow Tez task speculation to grant greater customization of certain parameters (Nishant Dash via jeagles)
TEZ-4034. Column selector filter should be case-insensitive (Jacob Tolar via jeagles)
TEZ-4043. Create a yetus compatible checkstyle configuration (Jonathan Eagles via kshukla)
TEZ-4041. TestExtServicesWithLocalMode fails in docker (Jonathan Eagles via kshukla)
TEZ-4040. Upgrade RoaringBitmap version to avoid NoSuchMethodError (Jonathan Eagles via kshukla)
TEZ-4037. Add back DAG search status KILLED (Jonathan Eagles via kshukla)
TEZ-4036. TestMockDAGAppMaster#testInternalPreemption should assert for failed state (Kuhu Shukla via jeagles)
TEZ-4028: Events not visible from proto history logging for s3a filesystem until dag completes (Harish JP, via Gopal V)
TEZ-4027. DagAwareYarnTaskScheduler can miscompute blocked vertices and cause a hang
TEZ-3957: Report TASK_DURATION_MILLIS as a Counter for completed tasks (Sergey Shelukhin, reviewed by Gopal V)
TEZ-4021. API incompatibility wro4j-maven-plugin
TEZ-4022. Upgrade Maven Surefire plugin to 3.0.0-M1
TEZ-3998. Allow CONCURRENT edge property in DAG construction and introduce ConcurrentSchedulingType (Yingda Chen via jeagles)
TEZ-4004. Update jetty9 to align with Hadoop and Hive (Jonathan Eagles via kshukla)
TEZ-4012. Add docker support for Tez. (Jonathan Eagles via kshukla)
TEZ-3976: Batch ShuffleManager error report events (Jaume Marhuenda, reviewed by Gopal V)
TEZ-3961. Tez UI web.xml tries to reach out to java.sun.com for validation after moving to jetty-9 (Kuhu Shukla via jeagles)
TEZ-3990. The number of shuffle penalties for a host/inputAttemptIdentifier should be capped (Kuhu Shukla via jeagles)
TEZ-4003. Add gopalv@apache.org to KEYS file (Gopal V via jeagles)
TEZ-3995. Fix dot files produced by tests to prevent ASF license warnings in yetus (Jaume Marhuenda via jegales)
TEZ-3969. TaskAttemptImpl: static fields initialized in instance ctor (Jaume Marhuenda via jegales)
TEZ-3994. Upgrade maven-surefire-plugin to 0.21.0 to support yetus
TEZ-3975. Add OWASP Dependency Check to the build
TEZ-3981. UnorderedPartitionedKVWriter.getInitialMemoryRequirement may return negative memory (Jaume M via jeagles)
TEZ-3982. DAGAppMaster and tasks should not report negative or invalid progress (Kuhu Shukla via jeagles)
TEZ-3972. Tez DAG can hang when a single task fails to fetch (Kuhu Shukla via jeagles)
TEZ-3988. Update snapshot version in master to 0.10.1-SNAPSHOT
TEZ-3989. Fix by-laws related to emeritus clause.
TEZ-3984: Shuffle: Out of Band DME event sending causes errors (Jaume Marhuenda, reviewed by Gopal V)
TEZ-3973. Add Kuhu Shukla's (kshukla) public key to KEYS (Kuhu Shukla via jeagles)
TEZ-3980: ShuffleRunner: the wake loop needs to check for shutdown (Gopal V, reviewed by Gunther Hagleitner)
TEZ-3958: Add internal vertex priority information into the tez dag.dot debug information (Jaume Marhuenda via Gopal V)
TEZ-3977. Add Eric Wohlstadter's public key to KEYS (Eric Wohlstadter via jeagles)
TEZ-3978. DAGClientServer Socket exception when localhost name lookup failures (Jonathan Eagles via jlowe)
TEZ-3974: Correctness regression of TEZ-955 in TEZ-2937 (Jaume Marhuenda, reviewed by Gopal V)
TEZ-3934. LegacySpeculator sometime issues wrong number of speculative attempts (Nishant Dash via jeagles)
TEZ-3942. RPC getTask writable optimization invalid in hadoop 2.8+
TEZ-3965: TestMROutput: Fix the hard-coded /tmp/output paths (Jaume Marhuenda, reviewed by Gopal V)
TEZ-3916: Add hadoop-azure-datalake jar to azure profile (Eric Wohlstadter via Gopal V)
TEZ-3970. NullPointerException in Tez ShuffleHandler Ranged Fetch (Jonathan Eagles via kshukla)
TEZ-3964. Inflater not closed in some places (Jaume M via jlowe)
TEZ-3955. Upgrade hadoop dependency to 3.0.3 (Jonathan Eagles via jlowe)
TEZ-3912. Fetchers should be more robust to corrupted inputs (Kuhu Shukla via jeagles)
TEZ-3963. Possible InflaterInputStream leaked in TezCommonUtils and related classes (Jaume M via jlowe)
TEZ-3954. Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords (Jonathan Eagles via kshukla)
TEZ-3960. Better error handling in proto history logger and add doAs support. (harishjp)
TEZ-3962. Configuration decode leaks an Inflater object (Eric Wohlstadter via jlowe)
TEZ-3959. HTTP 502 for bower install (Harish Jaiprakash via Sree)
TEZ-3953: Restore ABI-compat for DAGClient for TEZ-3951 (Sergey Shelukhin via Gopal V)
TEZ-3951. TezClient wait too long for the DAGClient for prewarm; tries to shut down the wrong DAG (Sergey Shelukhin via Harish Jaiprakash)
TEZ-3944. TestTaskScheduler times-out on Hadoop3 (Jonathan Eagles viak kshukla)
TEZ-3938. Task attempts failing due to not making progress (Kuhu Shukla via jeagles)
TEZ-3949. TestATSHistoryV15 is failing with hadoop3+ (Jonathan Eagles via kshukla)
TEZ-3946. NoClassDefFoundError, org.apache.hadoop.mapred.ShuffleHandler. (Multiple tests with Hadoop3) (Eric Wohlstadter via jeagles)
TEZ-3948. Tez distribution broken with hadoop3 (Jonathan Eagles via kshukla)
TEZ-3929. Upgrade Jersey to 1.19 (Eric Wohlstadter via jeagles)
TEZ-3947. TestATSHistoryWithACLs fails with Hadoop3 and Jersey 1.19 (Eric Wohlstadter via jeagles)
TEZ-3923. Move master to Hadoop 3+ and create separate 0.9.x line (Jonathan Eagles via kshukla)
TEZ-3943. TezClient leaks DAGClient for prewarm (Sergey Shelukhin via jlowe)
TEZ-3939. Remove performance hit of precondition check in AM for register running task attempt (Jonathan Eagles via jlowe)
TEZ-3940. Reduce time to convert TaskFinishedEvent to string (Jonathan Eagles via jlowe)
TEZ-3902. Upgrade to netty-3.10.5.Final.jar (Jason Lowe via jeagles)
TEZ-3935. DAG aware scheduler should release unassigned new containers rather than hold them (Jason Lowe via jeagles)
TEZ-3937. Empty partition BitSet to byte[] conversion creates one extra byte in rounding error (Jonathan Eagles via jlowe)
TEZ-3824. MRCombiner creates new JobConf copy per spill (Jonathan Eagles via jlowe)
TEZ-3933. Remove sleep from test TestExceptionPropagation (Jonathan Eagles via kshukla)
TEZ-3911: Optional min/max/avg aggr. task counters reported to HistoryLoggingService at final counter aggr (Vineet Garg, via Gopal V)
TEZ-3932. TaskSchedulerManager can throw NullPointerException during DAGAppMaster container cleanup race (Jonathan Eagles via jlowe)
TEZ-3931. TestExternalTezServices fails on Hadoop3 (Jonathan Eagles via kshukla)
TEZ-3930. TestDagAwareYarnTaskScheduler fails on Hadoop 3 (Jason Lowe via jeagles)
TEZ-3927. TestReduceProcessor fails on Hadoop 3.x (Jonathan Eagles via kshukla)
TEZ-3924. TestDefaultSorter fails intermittently due random keys and interaction with RLE and partition collisions (Jonathan Eagles via kshukla)
TEZ-3926. Changes to master for 0.10.x line and 0.9 release branch (jeagles)
Release 0.9.2: 2019-03-18
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-4052. Fit dot files ASF License issues - part 2 (Jonathan Eagles via kshukla)
TEZ-4044. Zookeeper: exclude jline from Zookeeper client from tez dist
TEZ-3995. Fix dot files produced by tests to prevent ASF license warnings in yetus (addendum)
TEZ-4031. Support tez gitbox migration (Jonathan Eagles via kshukla)
TEZ-4048. Make proto history logger queue size configurable
TEZ-4028: Events not visible from proto history logging for s3a filesystem until dag completes (Harish JP, via Gopal V)
TEZ-4047. Tez trademark in xml is causing xml parsing issue (Jonathan Eagles via kshukla)
TEZ-4050. maven site is failing due to missing configuration. (Jonathan Eagles via kshukla)
TEZ-4049. Fix findbugs issues in NotRunningJob (Jonathan Eagles via kshukla)
TEZ-4032. TEZ will throw Client cannot authenticate via:[TOKEN, KERBEROS] when used with HDFS federation(non viewfs, only hdfs schema used).
TEZ-4042. Speculative attempts should avoid running on the same node
TEZ-4035. Tez master breaks with YARN 3.2.0 ApplicationReport API change (jeagles)
TEZ-3952. Allow Tez task speculation to grant greater customization of certain parameters (Nishant Dash via jeagles)
TEZ-4034. Column selector filter should be case-insensitive (Jacob Tolar via jeagles)
TEZ-4043. Create a yetus compatible checkstyle configuration (Jonathan Eagles via kshukla)
TEZ-4004. Update jetty9 to align with Hadoop and Hive (Jonathan Eagles via kshukla)
TEZ-4041. TestExtServicesWithLocalMode fails in docker (Jonathan Eagles via kshukla)
TEZ-4040. Upgrade RoaringBitmap version to avoid NoSuchMethodError (Jonathan Eagles via kshukla)
TEZ-4037. Add back DAG search status KILLED (Jonathan Eagles via kshukla)
TEZ-4036. TestMockDAGAppMaster#testInternalPreemption should assert for failed state (Kuhu Shukla via jeagles)
TEZ-4027. DagAwareYarnTaskScheduler can miscompute blocked vertices and cause a hang
TEZ-4022. Upgrade Maven Surefire plugin to 3.0.0-M1
TEZ-4012. Add docker support for Tez. (Jonathan Eagles via kshukla)
TEZ-3961. Tez UI web.xml tries to reach out to java.sun.com for validation after moving to jetty-9 (Kuhu Shukla via jeagles)
TEZ-3995. Addendum (Jaume Marhuenda via jegales)
TEZ-3990. The number of shuffle penalties for a host/inputAttemptIdentifier should be capped (Kuhu Shukla via jeagles)
TEZ-4003. Add gopalv@apache.org to KEYS file (Gopal V via jeagles)
TEZ-3995. Fix dot files produced by tests to prevent ASF license warnings in yetus (Jaume Marhuenda via jegales)
TEZ-3969. TaskAttemptImpl: static fields initialized in instance ctor (Jaume Marhuenda via jegales)
TEZ-3994. Upgrade maven-surefire-plugin to 0.21.0 to support yetus
TEZ-3975. Add OWASP Dependency Check to the build
TEZ-3981. UnorderedPartitionedKVWriter.getInitialMemoryRequirement may return negative memory (Jaume M via jeagles)
TEZ-3982. DAGAppMaster and tasks should not report negative or invalid progress (Kuhu Shukla via jlowe)
Revert "TEZ-3982. DAGAppMaster and tasks should not report negative or invalid progress (Kuhu Shukla via jeagles)"
TEZ-3982. DAGAppMaster and tasks should not report negative or invalid progress (Kuhu Shukla via jeagles)
TEZ-3972. Tez DAG can hang when a single task fails to fetch (Kuhu Shukla via jeagles)
TEZ-3973. Add Kuhu Shukla's (kshukla) public key to KEYS (Kuhu Shukla via jeagles)
TEZ-3980: ShuffleRunner: the wake loop needs to check for shutdown (Gopal V, reviewed by Gunther Hagleitner)
TEZ-3977. Add Eric Wohlstadter's public key to KEYS (Eric Wohlstadter via jeagles)
TEZ-3978. DAGClientServer Socket exception when localhost name lookup failures (Jonathan Eagles via jlowe)
TEZ-3974: Correctness regression of TEZ-955 in TEZ-2937 (Jaume Marhuenda, reviewed by Gopal V)
TEZ-3934. LegacySpeculator sometime issues wrong number of speculative attempts (Nishant Dash via jeagles)
TEZ-3942. RPC getTask writable optimization invalid in hadoop 2.8+
TEZ-3970. NullPointerException in Tez ShuffleHandler Ranged Fetch (Jonathan Eagles via kshukla)
TEZ-3964. Inflater not closed in some places (Jaume M via jlowe)
TEZ-3912. Fetchers should be more robust to corrupted inputs (Kuhu Shukla via jeagles)
TEZ-3963. Possible InflaterInputStream leaked in TezCommonUtils and related classes (Jaume M via jlowe)
TEZ-3954. Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords (Jonathan Eagles via kshukla)
TEZ-3953: Restore ABI-compat for DAGClient for TEZ-3951 (Sergey Shelukhin via Gopal V)
TEZ-3960. Better error handling in proto history logger and add doAs support. (harishjp)
TEZ-3962. Configuration decode leaks an Inflater object (Eric Wohlstadter via jlowe)
TEZ-3959. HTTP 502 for bower install (Harish Jaiprakash via Sree)
TEZ-3951. TezClient wait too long for the DAGClient for prewarm; tries to shut down the wrong DAG (Sergey Shelukhin via Harish Jaiprakash)
TEZ-3938. Task attempts failing due to not making progress (Kuhu Shukla via jeagles)
TEZ-3947. TestATSHistoryWithACLs fails with Hadoop3 and Jersey 1.19 (Eric Wohlstadter via jeagles)
TEZ-3943. TezClient leaks DAGClient for prewarm (Sergey Shelukhin via jlowe)
TEZ-3939. Remove performance hit of precondition check in AM for register running task attempt (Jonathan Eagles via jlowe)
TEZ-3940. Reduce time to convert TaskFinishedEvent to string (Jonathan Eagles via jlowe)
TEZ-3935. DAG aware scheduler should release unassigned new containers rather than hold them (Jason Lowe via jeagles)
TEZ-3937. Empty partition BitSet to byte[] conversion creates one extra byte in rounding error (Jonathan Eagles via jlowe)
TEZ-3824. MRCombiner creates new JobConf copy per spill (Jonathan Eagles via jlowe)
TEZ-3933. Remove sleep from test TestExceptionPropagation (Jonathan Eagles via kshukla)
TEZ-3932. TaskSchedulerManager can throw NullPointerException during DAGAppMaster container cleanup race (Jonathan Eagles via jlowe)
TEZ-3931. TestExternalTezServices fails on Hadoop3 (Jonathan Eagles via kshukla)
TEZ-3930. TestDagAwareYarnTaskScheduler fails on Hadoop 3 (Jason Lowe via jeagles)
TEZ-3927. TestReduceProcessor fails on Hadoop 3.x (Jonathan Eagles via kshukla)
TEZ-3924. TestDefaultSorter fails intermittently due random keys and interaction with RLE and partition collisions (Jonathan Eagles via kshukla)
TEZ-3914. Recovering a large DAG fails to size limit exceeded (Jonathan Eagles via jlowe)
Revert "TEZ-3914. Recovering a large DAG fails to size limit exceeded (Jonathan Eagles via jlowe)"
TEZ-3873. A maven enforcer plugin dependency error in pom.xml (Jinjiang Ling via jeagles)
TEZ-3887. Tez Shuffle Handler should support Index Cache configuration (Jonathan Eagles via kshukla)
TEZ-3914. Recovering a large DAG fails to size limit exceeded (Jonathan Eagles via jlowe)
TEZ-3817. DAGs can hang after more than one uncaught Exception during doTransition. (kshukla)
TEZ-3915. Create protobuf based history event logger. (Harish Jaiprakash, reviewed by Gunther Hagleitner)
TEZ-3913. Precommit build fails to post to JIRA (Jason Lowe via jeagles)
Revert "TEZ-3913. Precommit build fails to post to JIRA (Jason Lowe via jeagles)"
TEZ-3913. Precommit build fails to post to JIRA (Jason Lowe via jeagles)
Revert "TEZ-3902. Upgrade to netty-3.10.5.Final.jar (Jason Lowe via kshukla)"
TEZ-3902. Upgrade to netty-3.10.5.Final.jar (Jason Lowe via kshukla)
TEZ-3909. DAG can hang if vertex with no tasks is killed (Jason Lowe via jeagles)
TEZ-3907. Improve log message to include the location the writers decide to spill output (Kuhu Shukla via jlowe)
TEZ-3905: Change BUILDING.TXT to minimum JDK 1.8 (Eric Wohlstadter via Gopal V)
TEZ-3874. NPE in TezClientUtils when "yarn.resourcemanager.zk-address" is present in Configuration. (Eric Wohlstadter via jlowe)
TEZ-3892: getClient API for TezClient (Eric Wohlstadter via Gopal V)
TEZ-3897. Tez Local Mode hang for vertices with broadcast input. (Jonathan Eagles via jlowe)
TEZ-3888: Update Jetty to org.eclipse.jetty 9.x (Eric Wohlstadter, reviewed by Rohini Palaniswamy)
TEZ-3898. TestTezCommonUtils fails when compiled against hadoop version >= 2.8 (Jason Lowe via jeagles)
TEZ-3896. TestATSV15HistoryLoggingService#testNonSessionDomains is failing (Jason Lowe via jeagles)
TEZ-3893. Tez Local Mode can hang for cases. (Jonathan Eagles via jlowe)
TEZ-3894. Tez intermediate outputs implicitly rely on permissive umask for shuffle (Jason Lowe via kshukla)
TEZ-3895. Missing name for local mode task scheduler service async request handler thread (Jonathan Eagles via kshukla)
TEZ-3770. DAG-aware YARN task scheduler (jlowe)
TEZ-3880: Do not count rejected tasks as killed in vertex progress (Sergey Shelukhin, reviewed by Gunther Hagleitner)
TEZ-3877. Delete unordered spill files once merge is done (Jason Lowe via jeagles)
TEZ-3883. Update version in master to 0.9.2 (zhiyuany)
TEZ-3882. Changes for 0.9.1 release (zhiyuany)
Release 0.9.1: 2017-11-16
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3876. Bug in local mode distributed cache files (Jacob Tolar via jeagles)
TEZ-3869. Analyzer: Fix VertexInfo::getLastTaskToFinish comparison
TEZ-3868. Update website to factor in the TEZ trademark registration
TEZ-3867. testSendCustomProcessorEvent try to get array out of read only ByteBuffer
TEZ-3855. Allow vertex manager to send event to processor
TEZ-3861. PipelineSorter setting negative progess
TEZ-3857. Tez TaskImpl can throw Invalid state transition for leaf tasks that do Retro Active Transition
TEZ-3862. Tez UI: Upgrade em-tgraph to version 0.0.14
TEZ-3858. Misleading dag level diagnostics in case of invalid vertex event
TEZ-3666. Integer overflow in ShuffleVertexManagerBase
TEZ-3856. API to access counters in InputInitializerContext
TEZ-3849. Combiner+PipelinedSorter silently drops records
TEZ-3853. Binary incompatibility caused by DEFAULT_LOG_LEVEL
TEZ-3854. Make use of new improved em-table sort-icon
TEZ-3850. Enable header as sort button on Tez UI
TEZ-3852. Optimize ContainerContext.isSuperSet to speed container reuse decisions
TEZ-3844. Tez UI Dag Counters show no records for a RUNNING DAG
TEZ-3848. Tez Local mode doesn't localize distributed cache files
TEZ-3847. AM web controller task counters are empty sometimes
TEZ-3830. HistoryEventTimelineConversion should not hard code the Task state
TEZ-3833. Tasks should report codec errors during shuffle as fetch failures
TEZ-3845. Tez UI Cleanup Stats Table
TEZ-3843. Tez UI Vertex/Tasks log links for running tasks are missing
TEZ-3836. Tez UI task page sort does not work on RHEL7/Fedora
TEZ-3840 addendum. Tez should write TEZ_DAG_ID before TEZ_EXTRA_INFO
TEZ-3840. Tez should write TEZ_DAG_ID before TEZ_EXTRA_INFO
TEZ-3839. Tez Shuffle Handler prints disk error stack traces for every read failure.
TEZ-3834. TaskSchedulerManager NullPointerException during shutdown when failed to start
TEZ-3724. Tez UI on HTTP corrects HTTPS REST calls to HTTP
TEZ-3831 addendum. Reduce Unordered memory needed for storing empty completed events
TEZ-3832. TEZ DAG status shows SUCCEEDED for SUCCEEDED_WITH_FAILURES final status
TEZ-3831. Reduce Unordered memory needed for storing empty completed events
TEZ-3828. Allow relaxing locality when retried task's priority is kept same
TEZ-3827. TEZ Vertex status on DAG index page shows SUCCEEDED for SUCCEEDED_WITH_FAILURES final status
TEZ-3825. Tez UI DAGs page can't query RUNNING or SUBMITTED apps
TEZ-3431. Add unit tests for container release
TEZ-3816. Add ability to automatically speculate single-task vertices
TEZ-3813. Reduce Object size of MemoryFetchedInput for large jobs
TEZ-3803. Tasks can get killed due to insufficient progress while waiting for shuffle inputs to complete
TEZ-3804. FetcherOrderedGrouped#setupLocalDiskFetch should ignore empty partition records
TEZ-3807. InMemoryWriter is not tested with RLE enabled
TEZ-3212. IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
TEZ-3752. Reduce Object size of InMemoryMapOutput for large jobs
TEZ-3805. Analyzer: Add an analyzer to find out scheduling misses in 1:1 edges
TEZ-3797. Add tez debug tool for comparing counters of 2 DAGs
Release 0.9.0: 2017-07-21
INCOMPATIBLE CHANGES
TEZ-3693. ControlledClock is not used.
TEZ-3745. Change master to required java 8
TEZ-3689. Change minimum hadoop version to 2.7.0.
TEZ-3611. Create lightweight summary events for ATS.
TEZ-3652. Remove ShuffleClientMetrics
TEZ-3659. AM/Task classpath should not contain hadoop conf directory.
ALL CHANGES:
TEZ-3798. Remove duplicate package-info.java
TEZ-3792. RootInputVertexManager doesn't drain queued source task completed events
TEZ-3795. Vertex state machine can throw InvalidStateTransitonException from TERMINATING state
TEZ-3794. tez-tools: swimlane does not recognize HistoryEventHandler.criticalEvents based logs
TEZ-3791. Failed/Killed task can throw InvalidStateTransitonException when a new attempt is launched
TEZ-3787. Remove Tez UI build and rebuild errors and warning due to yarn install and ember-truth-helpers
TEZ-3786. Fix Tez UI test failures after TEZ-3775
TEZ-3784. Submitting very large DAG throws com.google.protobuf.CodedInputStream exception
TEZ-3274. Vertex with MRInput and broadcast input does not respect slow start
TEZ-3775. Tez UI: Show DAG context in document title
TEZ-3605. Detect and prune empty partitions for the Ordered case
TEZ-3771. Tez UI: WASB/ADLS counters should be listed on the Tez UI
TEZ-3769. Unordered: Fix wrong stats being sent out in the last event, when final merge is disabled
TEZ-3777. Avoid buffer copies by passing RLE flag to TezMerger from PipelinedSorter
TEZ-3767. Shuffle should not report error to AM during inputContext.killSelf()
TEZ-3778. Remove SecurityInfo from tez-auxservices shaded jar
TEZ-3762. When final merge is disabled in unordered case, it should create index file instead of relying on cache
TEZ-3761. NPE in Fetcher under load
TEZ-3758. Vertex can hang in RUNNING state when two task attempts finish very closely and have retroactive failures
TEZ-3768. Test timeout value for TestShuffleHandlerJobs is low
TEZ-3766. Tez Aux-services : Clean up shaded jar to not include default config xml files and yarn-client pieces
TEZ-3760. Tez AUX Services: Shading needs to filter SIG files with -Pazure builds
TEZ-3741. Tez outputs should free memory when closed
TEZ-3698: UnorderedKV writer should be able to honor tez.runtime.enable.final-merge.in.output without pipelinedshuffle
TEZ-3732. Reduce Object size of InputAttemptIdentifier and MapOutput for large jobs
TEZ-3701. UnorderedPartitionedKVWriter - issues with parallel Deflater usage, synchronousqueue in threadpool
TEZ-3750. Add TEZ_RUNTIME_UNORDERED_PARTITIONED_KVWRITER_BUFFER_MERGE_PERCENT to UnorderedPartitionedKVOutput
TEZ-3749. Get map and reduce task memory from JobConf
TEZ-3748. TaskAttemptImpl State Machine Invalid event: TA_SUBMITTED at KILL_IN_PROGRESS
TEZ-3739. Fair CartesianProduct doesn't works well with huge difference in output size
TEZ-3714. Tez UI: Hive Queries page: Use Dag ID and App ID if they are published form Hive side
TEZ-3747. TezConstants.TEZ_SHUFFLE_HANDLER_SERVICE_ID is referenced in Hive
TEZ-3736. SubmittedDAGs is always 0 in Resource Manager UI.
TEZ-3744. Fix findbugs warnings after TEZ-3334 merge
TEZ-3743. TestTaskCommunicatorContextImpl throws NullPointerException after TEZ-3334 merge
TEZ-3742. Fix AMContainerHelpers#createCommonContainerLaunchContext to not pass localResources.
TEZ-3737. FairCartesianProductVertexMananger used incorrect #partition
TEZ-3691. Setup fetchers to use shared executor
TEZ-3662. Vertex Duration in 0.9 Tez UI regression from 0.7
TEZ-3730. Lower logging level in UnorderedPartitionedKVWriter.
TEZ-3723. TezIndexRecord#hasData() returns true for empty index record in the Unordered case
TEZ-3716. Allow attempt retries to be treated the same as the first attempt.
TEZ-3715. Differentiate between TaskAttempt submission and TaskAttempt started.
TEZ-3708. Improve parallelism and auto grouping of unpartitioned cartesian product
TEZ-3717. tez-yarn-timeline-history-with-fs does not build with hadoop-2.8.
TEZ-3697. Adding #output_record in vertex manager event payload
TEZ-3673. Allocate smaller buffers in UnorderedPartitionedKVWriter.
TEZ-3707. TezSharedExecutor race condition in awaitTermination vs isTerminated.
TEZ-3700. Consumer attempt should kill itself instead of failing during validation checks with final merge avoidance
TEZ-3703. Use a sha comparison to compare vertex and dag resources, if there is a mismatch.
TEZ-3699. For large dataset, pipelined shuffle throws exceptions in consumer side for UnorderedPartitioned edge
TEZ-3695. TestTezSharedExecutor fails sporadically.
TEZ-2049. Remove YARN references from Tez AsyncDispatcher
TEZ-3675. Handle changes to ResourceCalculatorProcessTree in YARN-3427 for Hadoop 3.x
TEZ-3690. Tez on hadoop 3 build failed due to hdfs client/server jar separation
TEZ-3687. Code smell in DAGStatus and VertexStatus equals implementation
TEZ-3631. Tez UI: TEZ_DAG_EXTRA_INFO compatibility changes - Makes All DAGs page faster
TEZ-3680. Optimizations to UnorderedPartitionedKVWriter
TEZ-1187. Add a framework ExecutorService which shares threads
TEZ-3654. Make CartesianProduct edge work with GroupInputEdge
TEZ-3285. Tez UI: Lock down dependency versions
TEZ-3683. LocalContainerLauncher#shouldDelete member variable is not used
TEZ-3681. Improve UI error message while trying to sort running DAGs with Auto Refresh enabled
TEZ-3668. Explicitly include hadoop-mapreduce-client-shuffle for test in root pom
TEZ-3667. Stop using org.apache.hadoop.security.ssl.SSLFactory.DEFAULT_SSL_REQUIRE_CLIENT_CERT
TEZ-3665. TestATSV15HistoryLoggingService should use mocked TimelineClient
TEZ-1526. LoadingCache for TezTaskID slow for large jobs
TEZ-3653. Tez UI: Swimlane tooltip is not proper for running DAGs
TEZ-3656. Tez UI: Status correction is not working as expected
TEZ-3650. Improve performance of FetchStatsLogger#logIndividualFetchComplete
TEZ-3655. Specify netty version instead of inheriting from hadoop dependency.
TEZ-3503. Tez UI: Support search by queue name
TEZ-3642. Tez UI: Auto-refresh is not stopping when DAG is the main entity
TEZ-3253. Remove special handling for last app attempt.
TEZ-3648. IFile.Write#close has an extra output stream flush
TEZ-3649. AsyncHttpConnection should add StopWatch start
TEZ-3647. Add a setting which lets Tez determine Xmx.
TEZ-3646. IFile.Writer has an extra output stream flush call
TEZ-3640. Tez UI: Add associated llap application id to queries page
TEZ-3639. Tez UI: Footer pagination is improper in landing page
TEZ-3637. TezMerger logs too much at INFO level.
TEZ-3638. VertexImpl logs too much at info when removing tasks after auto-reduce parallelism
TEZ-3630. Tez UI: Use DAG status for controlling auto-refresh polling
TEZ-3634. reduce the default buffer sizes in PipelinedSorter by a small amount.
TEZ-3626. Tez UI: First Task Start Time & Last Task Finish Time values are showing up incorrectly
TEZ-3629. Tez UI: Enable the UI to display log links from LLAP
TEZ-3627. Use queue name available in RegisterApplicationMasterResponse for publishing to ATS.
TEZ-3610. TEZ UI 0.7 0.9 compatibility for url query params and tez-app sub-routes
TEZ-3619. Tez UI: Improve DAG Data download
TEZ-3615. Tez UI: Table changes
TEZ-3267. Publish queue name to ATS as part of dag summary.
TEZ-3602. Tez UI: Query Name field is not required
TEZ-3581. Add different logger to enable suppressing logs for specific lines.
TEZ-3600. Fix flaky test: TestTokenCache.
TEZ-3598. Tez UI: Text formatting changes
TEZ-3594. Tez UI: Graphical view tooltip issues
TEZ-3593. Tez UI: Issues in timeline page
TEZ-3592. Tez UI: Search issues
TEZ-3591. Tez UI: Logs url in all DAGs doesn't open in a new window
TEZ-3589. add a unit test for amKeepAlive not being shutdown if an app takes a long time to launch.
TEZ-3554. Add a link to get to all logs from Tez UI while job is running
TEZ-3417. Reduce sleep time on AM shutdown to reduce test runtimes
TEZ-3494. Support relative url for tez-ui.history-url.base config
TEZ-3575. RM have started forwarding origin. Use that in AMWebController for CORS support
TEZ-3580. Tez UI: Pagination broken on queries page
TEZ-3584. amKeepAliveService in TezClient should shutdown in case of AM failure.
TEZ-3583. Tez UI: UTs are flaky because of a dependency issue
TEZ-3579. Wrong configuration key for max slow start fraction in CartesianProductVertexManager.
TEZ-2712. Tez UI: Display the vertex description in the tooltip of vertex in DAG view UI
TEZ-3571. Tez UI: Display a Total Timeline View for Hive Queries
TEZ-3496. Tez UI: Optimize display of all tasks table
TEZ-3556. Tez UI: Display query configurations
TEZ-3531. Tez UI: All Queries table: Improve searchability
TEZ-3530. Tez UI: Add query details page, and link the page from All Queries table
TEZ-3529. Tez UI: Add 'All Queries' table in the landing page along 'All DAGs' page
TEZ-3458. Auto grouping for cartesian product edge(unpartitioned case).
TEZ-3443. Remove a repeated/unused method from MRTask.
TEZ-3551: FrameworkClient created twice causing minor delay
TEZ-3504. Tez UI: Duration is displaying invalid values when start or end time is invalid
TEZ-3570. Tez UI: Wait for sometime before tooltips are displayed
TEZ-3555. Tez UI: Build is failing in RHEL6
TEZ-3565: amConfig should check queuename isEmpty
TEZ-3558. CartesianProduct is missing from the ExampleDriver class
TEZ-3552. Shuffle split array when size-based sorting is turned off.
TEZ-3271. Provide mapreduce failures.maxpercent equivalent.
TEZ-3222. Reduce messaging overhead for auto-reduce parallelism case
TEZ-3547. Add TaskAssignment Analyzer
TEZ-3508. TestTaskScheduler cleanup.
TEZ-3269. Provide basic fair routing and scheduling functionality via custom VertexManager and EdgeManager.
TEZ-3477. MRInputHelpers generateInputSplitsToMem public API modified
TEZ-3465. Support broadcast edge into cartesian product vertex and forbid other edges.
TEZ-3502. Tez UI: Search in All DAGs page doesn't work with numeric values
TEZ-3470. Tez UI: Make the build work in IBM PPC
TEZ-3457. Add more unit test coverage for container reuse.
TEZ-3215. Support for MultipleOutputs.
TEZ-3484. Tez UI: Remove .travis.yml from webapp folder
TEZ-3405. Support ability for AM to kill itself if there is no client heartbeating to it.
TEZ-3469. Tez UI: Bump Phantom JS version to 2.1.1
TEZ-3430. Make split sorting optional.
TEZ-3466. Tez classpath building to mimic mapreduce classpath building
TEZ-3428. Tez UI: First Tab not needed for few entries in DAG listings
TEZ-3453. Correct the downloaded ATS dag data location for analyzer
TEZ-3449. Fix Spelling typos.
TEZ-3433. Tez UI: Searching using wrong ID causes error in all DAGs page
TEZ-3429. Set reconfigureDoneTime on VertexConfigurationDoneEvent properly.
TEZ-3163. Reuse and tune Inflaters and Deflaters to speed DME processing
TEZ-3434. Add unit tests for flushing of recovery events.
TEZ-3404. Move blocking call for YARN Timeline domain creation from client side to AM.
TEZ-3272. Add AMContainerImpl and AMNodeImpl to StateMachine visualization list.
TEZ-3284. Synchronization for every write in UnorderdKVWriter
TEZ-3230. Implement vertex manager and edge manager of cartesian product edge.
TEZ-3395. Refactor ShuffleVertexManager to make parts of it re-usable in other plugins.
TEZ-3382. Tez analyzer: Should be resilient to new counters
TEZ-3379. Tez analyzer: Move sysout to log4j
TEZ-3333. Tez UI: Handle cases where Vertex/Task/Task Attempt data is missing
TEZ-3303. Have ShuffleVertexManager consume more precise partition stats.
TEZ-3329. Tez ATS data is incomplete for a vertex which fails or gets killed before initialization
TEZ-3327. ATS Parser: Populate config details available in dag
TEZ-3325. Flaky test in TestDAGImpl.testCounterLimits.
TEZ-3313. ATSFileParser : Wrong args passed in VersionInfo
TEZ-3288. Tez UI: Display more details in the error bar
TEZ-3216. Add support for more precise partition stats in VertexManagerEvent.
TEZ-3295. TestOrderedWordCount should handle relative input/output paths.
TEZ-3292. Tez UI: UTs breaking with timezone change
TEZ-2846. Flaky test: TestCommit.testVertexCommit_OnDAGSuccess.
TEZ-3264. Tez UI: UI discrepancies
TEZ-3289. Tez Example MRRSleep job does not set Staging dir correctly on secure cluster.
TEZ-3276. Tez Example MRRSleep job fails when tez.staging-dir fs is not same as default FS.
TEZ-3063. Tez UI: Display Input, Output, Processor, Source and Sink configurations under a vertex
TEZ-3206. Have unordered partitioned KV output send partition stats via VertexManagerEvent.
TEZ-3255. Tez UI: Hide swimlane while displaying running DAGs from old versions of Tez
TEZ-3254. Tez UI: Consider downloading Hive/Pig explain plans
TEZ-3086. Tez UI: Backward compatibility changes
TEZ-3245. Data race between addKnowInput and clearAndGetOnepartition of InputHost.
TEZ-3193. Deadlock in AM during task commit request.
TEZ-3233. Tez UI: Have LLAP information reflect in Tez UI
TEZ-3203. DAG hangs when one of the upstream vertices has zero tasks
TEZ-3207. Add support for fetching multiple partitions from the same source task to UnorderedKVInput.
TEZ-3232. Disable randomFailingInputs in testFaulttolerance to unblock other tests.
TEZ-3227. Tez UI: Replace UI1 with UI2
TEZ-3228. Update version in master to 0.9.0.
TEZ-3334. Tez Custom Shuffle Handler:
TEZ-3713. Allow dag level deletion in cases where containers are reused
TEZ-3712. Use Local FileContext for deleting dag level directories
TEZ-3633. Implement keep-alive timeout in tez shuffle handler
TEZ-3740. Clean up TEZ-3334-CHANGES.txt
TEZ-3735. Test failures in TestTaskAttempt and TestAMContainerMap
TEZ-3726. Clean up DeletionTracker's reflection instantiation and provide ContainerLauncher with dagComplete() functionality
TEZ-3725. Cleanup http connections and other unnecessary fields in DAG Deletion tracker classes.
TEZ-3705. Modify DeletionTracker and deletion threads to be initialized only if enabled for tez_shuffle
TEZ-3702. Tez shuffle jar includes service loader entry for ClientProtocolProvider but not the corresponding class
TEZ-3685. ShuffleHandler completedInputSet off-by-one error
TEZ-3684. Incorporate first pass non-essential TEZ-3334 pre-merge feedback
TEZ-3682. Pass parameters instead of configuration for changes to support tez shuffle handler
TEZ-3628. Give Tez shuffle handler threads custom names
TEZ-3621. Optimize the Shuffle Handler content length calculation for keep alive
TEZ-3620. UnorderedPartitionedKVOutput is missing the shuffle service config in the confKeys set
TEZ-3618. Shuffle Handler Loading cache equality tests always results is false
TEZ-3612. Tez Shuffle Handler Content length does not match actual
TEZ-3608. Fetcher can hang if copyMapOutput/fetchInputs returns early
TEZ-3606. Fix debug log for empty partitions to the expanded partitionId in the Composite case
TEZ-3604. Remove the compositeInputAttemptIdentifier from remaining list upon fetch completion in the Ordered case
TEZ-3599. Unordered Fetcher can hang if empty partitions are present
TEZ-3596. Number of Empty DME logged for Composite fetch is too high
TEZ-3597. Composite Fetch hangs on certain DME empty events.
TEZ-3595. Composite Fetch account error for disk direct
TEZ-3590. Remove google.protobuf from the tez-auxservices shaded jar
TEZ-3587. Fetcher fetchInputs() can NPE on srcAttempt due to missing entry in pathToAttemptMap
TEZ-3586. Remove fusesource.leveldbjni from the tez-auxservices shaded jar
TEZ-3532. Backport MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log
TEZ-3563. Tez Shuffle Handler logging fails to initialize
TEZ-3564. TezConfiguration#TEZ_DELETION_TRACKER_CLASS has the wrong config key-name
TEZ-3557. TEZ-3362 causes TestContainerLauncherWrapper#testDelegation to fail
TEZ-3361. Fetch Multiple Partitions from the Shuffle Handler
TEZ-3509. Make DAG Deletion path based
TEZ-3480. Port MAPREDUCE-6763 to Tez ShuffleHandler
TEZ-3362. Delete intermediate data at DAG level for Shuffle Handler
TEZ-3360. Tez Custom Shuffle Handler Documentation
TEZ-3411. TestShuffleHandler#testSendMapCount should not used hard coded ShuffleHandler port
TEZ-3412. Modify ShuffleHandler to use Constants.DAG_PREFIX and fix AttemptPathIdentifier#toString()
TEZ-3410. ShuffleHandler should use Path.SEPARATOR instead of '/'
TEZ-3408. Allow Task Output Files to reside in DAG specific directories for Custom Shuffle Handler
TEZ-3238. Shuffle service name should be configureable and should not be hardcoded to ‘mapreduce_shuffle’
TEZ-3390. Package Shuffle Handler as a shaded uber-jar
TEZ-3393. Remove extra jetty dependency from Shuffle Handler
TEZ-3378. Move Shuffle Handler configuration into the Tez namespace
TEZ-3377. Remove ShuffleHandler dependency on mapred.FadvisedChunkedFile and mapred.FadvisedFileRegion
TEZ-3380. Shuffle Handler: Replace primitive wrapper's valueOf method with parse* method to avoid unnecessary boxing/unboxing
TEZ-3355. Tez Custom Shuffle Handler POC
Release 0.8.6: Unreleased
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3007. Use AppFinalState.ENDED when unregistering with the RM in session mode
TEZ-3679. Minor ASF header issues.
TEZ-3678. The command "hadoop dfs" should be replaced by "hadoop fs" in install markdown.
TEZ-3677. by-laws markdown has an incorrect license header.
TEZ-3671. TestCompositeDataMovementEvent has a misplaced Apache license header.
TEZ-3672. Remove duplicate Apache license headers.
TEZ-3561. Fix wrong tez tarball name in install.md.
Release 0.8.5: 2016-03-13
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3709. TezMerger is slow for high number of segments
TEZ-3719. DAGImpl.computeProgress slows down dispatcher and ipc threads
TEZ-3616. TestMergeManager#testLocalDiskMergeMultipleTasks fails intermittently
TEZ-3644. Cleanup container list stored in AMNode.
TEZ-3643. Long running AMs can go out of memory due to retained AMContainer instances.
TEZ-3624. Split multiple calls on the same line in TaskCommunicatorContextImpl.
TEZ-3550. Provide access to sessionId/dagId via DagClient.
TEZ-3609. Improve ATSv15 performance for DAG entities read calls.
TEZ-3244. Allow overlap of input and output memory when they are not concurrent
TEZ-3601. Add another HistoryLogLevel to suppress TaskAttempts at specific levels
TEZ-3582. Exception swallowed in PipelinedSorter causing incorrect results.
TEZ-3462. Task attempt failure during container shutdown loses useful container diagnostics
TEZ-3574. Container reuse won't pickup extra dag level local resource.
TEZ-3566. Avoid caching fs isntances in TokenCache after a point.
TEZ-3568. Update SecurityUtils configuration to pick user provided configuration.
TEZ-3559. TEZ_LIB_URIS doesn't work with schemes different than the defaultFS
TEZ-3549. TaskAttemptImpl does not initialize TEZ_TASK_PROGRESS_STUCK_INTERVAL_MS correctly
TEZ-3537. ArrayIndexOutOfBoundsException with empty environment variables/Port YARN-3768 to Tez
TEZ-3507. Task logs link when editing url from one task to another.
TEZ-3536. NPE in WebUIService start when host resolution fails.
TEZ-3534. Differentiate thread names on Fetchers, minor changes to shuffle shutdown code.
TEZ-3491. Tez job can hang due to container priority inversion.
TEZ-3533. ShuffleScheduler should shutdown threadpool on exit.
TEZ-3493. DAG submit timeout cannot be set to a month
TEZ-3505. Move license to the file header for TezBytesWritableSerialization
TEZ-3486. COMBINE_OUTPUT_RECORDS/COMBINE_INPUT_RECORDS are not correct
TEZ-3097. Flaky test: TestCommit.testDAGCommitStartedEventFail_OnDAGSuccess.
TEZ-3487. Improvements in travis yml file to get builds to work.
TEZ-3483. Create basic travis yml file for Tez.
TEZ-3437. Improve synchronization and the progress report behavior for Inputs from TEZ-3317.
TEZ-3317. Speculative execution starts too early due to 0 progress.
TEZ-3452. Auto-reduce parallelism calculation can overflow with large inputs
TEZ-3439. Tez joinvalidate fails when first input argument size is bigger than the second.
TEZ-3464. Fix findbugs warnings in tez-dag mainLoop
TEZ-3330. Propagate additional config parameters when running MR jobs via Tez.
TEZ-3335. DAG client thinks app is still running when app status is null
TEZ-3460. Fix precommit release audit warning
TEZ-3368. NPE in DelayedContainerManager
TEZ-3440. Shuffling to memory can get out-of-sync when fetching multiple compressed map outputs
TEZ-3429. Set reconfigureDoneTime on VertexConfigurationDoneEvent properly.
TEZ-3000. Fix TestContainerReuse.
TEZ-3436. Check input and output count before start in MapProcessor.
TEZ-3426. Second AM attempt launched for session mode and recovery disabled for certain cases
TEZ-3326. Display JVM system properties in AM and task logs.
TEZ-3009. Errors that occur during container task acquisition are not logged.
TEZ-2852. TestVertexImpl fails due to race in AsyncDispatcher.
TEZ-3413. ConcurrentModificationException in HistoryEventTimelineConversion for AppLaunchedEvent.
TEZ-3352. MRInputHelpers getStringProperty() should not fail if property value is null.
TEZ-3409. Log dagId along with other information when submitting a dag.
TEZ-3384. Fix TestATSV15HistoryLoggingService::testDAGGroupingGroupingEnabled unit test.
TEZ-3376. Fix groupId generation to account for dagId starting with 1.
TEZ-3359. Add granular log levels for HistoryLoggingService.
TEZ-3374. Change TEZ_HISTORY_LOGGING_TIMELINE_NUM_DAGS_PER_GROUP conf key name.
TEZ-3358. Support using the same TimelineGroupId for multiple DAGs.
TEZ-3357. Change TimelineCachePlugin to handle DAG grouping.
TEZ-3348. NullPointerException in Tez MROutput while trying to write using Parquet's DeprecatedParquetOutputFormat.
TEZ-3356. Fix initializing of stats when custom ShuffleVertexManager is used.
TEZ-3329. Tez ATS data is incomplete for a vertex which fails or gets killed before initialization.
TEZ-3235. Modify Example TestOrderedWordCount job to test the IPC limit for large dag plans.
TEZ-3337. Do not log empty fields of TaskAttemptFinishedEvent to avoid confusion.
TEZ-1248. Reduce slow-start should special case 1 reducer runs.
TEZ-3370. Tez UI: Display the log link as N/A if the app does not provide a log link
TEZ-3398. Tez UI: Bread crumb link to Application from Application details dag/configuration tab is broken
TEZ-3433. Tez UI: Searching using wrong ID causes error in all DAGs page
TEZ-3419. Tez UI: Applications page shows error, for users with only DAG level ACL permission
TEZ-3347. Tez UI: Vertex UI throws an error while getting vertexProgress for a killed Vertex
TEZ-3546. Tez UI: On sorting asc - Not Available must be at the top
Release 0.8.4: 2016-07-08
INCOMPATIBLE CHANGES
ALL CHANGES:
TEZ-3323. Update licese and notice for xml-apis jar. Also update year in notice to 2016.
TEZ-3223. Support a NullHistoryLogger to disable history logging if needed.
TEZ-3286. Allow clients to set processor reserved memory per vertex (instead of per container).
TEZ-3293. Fetch failures can cause a shuffle hang waiting for memory merge that never starts.
TEZ-3314. Double counting input bytes in MultiMRInput.
TEZ-3308. Add counters to capture input split length.
TEZ-3302. Add a version of processorContext.waitForAllInputsReady and waitForAnyInputReady with a timeout.
TEZ-3291. Optimize splits grouping when locality information is not available.
TEZ-3305. TestAnalyzer fails on Hadoop 2.7.
TEZ-3304. TestHistoryParser fails with Hadoop 2.7.
TEZ-3296. Tez job can hang if two vertices at the same root distance have different task requirements
TEZ-3294. DAG.createDag() does not clear local state on repeat calls.
TEZ-3297. Deadlock scenario in AM during ShuffleVertexManager auto reduce.
TEZ-3296. Tez fails to compile against hadoop 2.8 after MAPREDUCE-5870
TEZ-3290. Set full task attempt id string in MRInput configuration object.
TEZ-3278. Hide Swimlane from Tez UI
TEZ-3280. LOG MRInputHelpers split generation message as INFO
TEZ-909. Provide support for application tags
TEZ-2769. TEZ-UI Hive SQL is only displayed to line 11
TEZ-3257. Fix flaky test TestUnorderedPartitionedKVWriter.
TEZ-3240. Improvements to tez.lib.uris to allow for multiple tarballs and mixing tarballs and jars.
TEZ-3237. Corrupted shuffle transfers to disk are not detected during transfer
TEZ-3246. Improve diagnostics when DAG killed by user
TEZ-3258. Jvm Checker does not ignore DisableExplicitGC when checking JVM GC options.
TEZ-3256. [Backport HADOOP-11032] Remove Guava Stopwatch dependency
TEZ-2342. Reduce bytearray copy with TezEvent Serialization and deserialization
TEZ-3251. Allow ability to add custom counters to TaskRunner2Callable.
TEZ-3250. TezTaskRunner2 should accept ExecutorService.
TEZ-3193. Deadlock in AM during task commit request.
TEZ-3203. DAG hangs when one of the upstream vertices has zero tasks
TEZ-3219. Allow service plugins to define log locations link for remotely run task attempts.
TEZ-3224. User payload is not initialized before creating vertex manager plugin.
TEZ-3226. Tez UI 2: All DAGs UX improvements.
TEZ-3077. TezClient.waitTillReady should support timeout.
TEZ-3202. Reduce the memory need for jobs with high number of segments
TEZ-3165. Allow Inputs/Outputs to be initialized serially, control processor initialization relative to Inputs/Outputs
TEZ-3214. Tez UI 2: Pagination in All DAGs
TEZ-3210. Tez UI 2: license should account for numeral, more-js, loader.js , etc
TEZ-3087. Tez UI 2: Add log links in task & attempt details page
TEZ-3146. Tez UI 2: CSS & JS assets in the UI must be minified
TEZ-3259. Tez UI: Build issue - File saver package is not working well with bower
TEZ-3262. Tez UI : zip.js is not having a bower friendly versioning system
TEZ-3281. Tez UI: Swimlane improvements
TEZ-3318. Tez UI: Polling is not restarted after RM recovery
Release 0.8.3: 2016-04-14
INCOMPATIBLE CHANGES
TEZ-3180. Update master docs to declare hadoop-2.6.x as a minimum requirement.
TEZ-3029. Add an onError method to service plugin contexts.
TEZ-3120. Remove TaskCommContext.getCurrentDagName, Identifier.
TEZ-3183. Change the taskFailed method on plugin contexts to specify the type of failure.
TEZ-3199. Rename getCredentials in TaskCommunicatorContext to be less confusing.
ALL CHANGES:
TEZ-3188. Move tez.submit.hosts out of TezConfiguration to TezConfigurationConstants.
TEZ-3194. Tez UI: Swimlane improve in-progress experience.
TEZ-3196. java.lang.InternalError from decompression codec is fatal to a task during shuffle
TEZ-3161. Allow task to report different kinds of errors - fatal / kill.
TEZ-3177. Non-DAG events should use the session domain or no domain if the data does not need protection.
TEZ-3192. IFile#checkState creating unnecessary objects though auto-boxing
TEZ-3173. Update Tez AM REST APIs for more information for each vertex.
TEZ-3108. Add support for external services to local mode.
TEZ-3189. Pre-warm dags should not be counted in submitted dags count by DAGAppMaster.
TEZ-2967. Vertex start time should be that of first task start time in UI
TEZ-3175. Add tez client submit host
TEZ-3166. Fix a few cases where counters aren't fully updated and sent for failed tasks.
TEZ-2958. Recovered TA, whose commit cannot be recovered, should move to killed state
TEZ-2936. Create ATS implementation that enables support for YARN-4265 (ATS v1.5)
TEZ-3148. Invalid event TA_TEZ_EVENT_UPDATE on TaskAttempt.
TEZ-3105. TezMxBeanResourceCalculator does not work on IBM JDK 7 or 8 causing Tez failures.
TEZ-3155. Support a way to submit DAGs to a session where the DAG plan exceeds hadoop ipc limits.
TEZ-2863. Container, node, and logs not available in UI for tasks that fail to launch
TEZ-3140. Reduce AM memory usage during serialization
TEZ-2756. MergeManager close should not try merging files on close if invoked after a shuffle exception.
TEZ-3156. Tez client keeps trying to talk to RM even if RM does not know about the application.
TEZ-3115. Shuffle string handling adds significant memory overhead
TEZ-3151. Expose DAG credentials to plugins.
TEZ-3149. Tez-tools: Add username in DagInfo.
TEZ-2988. DAGAppMaster::shutdownTezAM should return with a no-op if it has been invoked earlier.
TEZ-3147. Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory
TEZ-3141. mapreduce.task.timeout is not translated to container heartbeat timeout
TEZ-3128. Avoid stopping containers on the AM shutdown thread.
TEZ-3129. Tez task and task attempt UI needs application fails with NotFoundException
TEZ-3114. Shuffle OOM due to EventMetaData flood
TEZ-1911. MergeManager's unconditionalReserve() should check for memory limits before allocating.
TEZ-3102. Fetch failure of a speculated task causes job hang
TEZ-3124. Running task hangs due to missing event to initialize input in recovery.
TEZ-3135. tez-ext-service-tests, tez-plugins/tez-yarn-timeline-history and tez-tools/tez-javadoc-tools missing dependencies.
TEZ-3134. tez-dag should depend on commons-collections4.
TEZ-3126. Log reason for not reducing parallelism
TEZ-3131. Support a way to override test_root_dir for FaultToleranceTestRunner.
TEZ-3067. Links to tez configs documentation should be bubbled up to top-level release page.
TEZ-3123. Containers can get re-used even with conflicting local resources.
TEZ-3117. Deadlock in Edge and Vertex code
TEZ-3103. Shuffle can hang when memory to memory merging enabled
TEZ-3107. tez-tools: Log warn msgs in case ATS has wrong values (e.g startTime > finishTime).
TEZ-3104. Tez fails on Bzip2 intermediate output format on hadoop 2.7.1 and earlier
TEZ-3090. MRInput should make dagIdentifier, vertexIdentifier, etc available to the InputFormat jobConf.
TEZ-3093. CriticalPathAnalyzer should be accessible via zeppelin.
TEZ-3089. TaskConcurrencyAnalyzer can return negative task count with very large jobs.
TEZ-2307. Possible wrong error message when submitting new dag
TEZ-2974. Tez tools: TFileRecordReader in tez-tools should support reading >2 GB tfiles.
TEZ-3081. Update tez website for trademarks feedback.
TEZ-3076. Reduce merge memory overhead to support large number of in-memory mapoutputs
TEZ-3079. Fix tez-tfile parser documentation.
TEZ-3066. TaskAttemptFinishedEvent ConcurrentModificationException in recovery or history logging services.