Talk about ascendingtimestampextractor of flynk

Time:2021-10-22

order

This paper mainly studies the ascendingtimestampextractor of Flink

AscendingTimestampExtractor

flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/functions/timestamps/AscendingTimestampExtractor.java

/**
 * A timestamp assigner and watermark generator for streams where timestamps are monotonously
 * ascending. In this case, the local watermarks for the streams are easy to generate, because
 * they strictly follow the timestamps.
 *
 * @param <T> The type of the elements that this function can extract timestamps from
 */
@PublicEvolving
public abstract class AscendingTimestampExtractor<T> implements AssignerWithPeriodicWatermarks<T> {

    private static final long serialVersionUID = 1L;

    /** The current timestamp. */
    private long currentTimestamp = Long.MIN_VALUE;

    /** Handler that is called when timestamp monotony is violated. */
    private MonotonyViolationHandler violationHandler = new LoggingHandler();


    /**
     * Extracts the timestamp from the given element. The timestamp must be monotonically increasing.
     *
     * @param element The element that the timestamp is extracted from.
     * @return The new timestamp.
     */
    public abstract long extractAscendingTimestamp(T element);

    /**
     * Sets the handler for violations to the ascending timestamp order.
     *
     * @param handler The violation handler to use.
     * @return This extractor.
     */
    public AscendingTimestampExtractor<T> withViolationHandler(MonotonyViolationHandler handler) {
        this.violationHandler = requireNonNull(handler);
        return this;
    }

    // ------------------------------------------------------------------------

    @Override
    public final long extractTimestamp(T element, long elementPrevTimestamp) {
        final long newTimestamp = extractAscendingTimestamp(element);
        if (newTimestamp >= this.currentTimestamp) {
            this.currentTimestamp = newTimestamp;
            return newTimestamp;
        } else {
            violationHandler.handleViolation(newTimestamp, this.currentTimestamp);
            return newTimestamp;
        }
    }

    @Override
    public final Watermark getCurrentWatermark() {
        return new Watermark(currentTimestamp == Long.MIN_VALUE ? Long.MIN_VALUE : currentTimestamp - 1);
    }

    //......
}
  • The ascendingtimestampextractor abstract class implements the extracttimestamp and getcurrentwatermark methods of the assignerwithperiodicwatermarks interface, and declares the abstract method extractascendingtimestamp for subclass implementation
  • Ascendingtimestampextractor is applicable to the time of elements, which is monotonically increasing in each parallel task(timestamp monotony)In the scenario of extracttimestamp, first call the extractascendingtimestamp method implemented by the subclass to extract the new timestamp from the element, and then return. For those that violate the timestamp monotony, call the monotonyviolationhandler to handle it
  • Getcurrentwatermark method is not long.min in currenttimestamp_ Return watermark when value (currenttimestamp – 1)

MonotonyViolationHandler

flink-streaming-java_2.11-1.7.0-sources.jar!/org/apache/flink/streaming/api/functions/timestamps/AscendingTimestampExtractor.java

    /**
     * Interface for handlers that handle violations of the monotonous ascending timestamps
     * property.
     */
    public interface MonotonyViolationHandler extends java.io.Serializable {

        /**
         * Called when the property of monotonously ascending timestamps is violated, i.e.,
         * when {@code elementTimestamp < lastTimestamp}.
         *
         * @param elementTimestamp The timestamp of the current element.
         * @param lastTimestamp The last timestamp.
         */
        void handleViolation(long elementTimestamp, long lastTimestamp);
    }

    /**
     * Handler that does nothing when timestamp monotony is violated.
     */
    public static final class IgnoringHandler implements MonotonyViolationHandler {
        private static final long serialVersionUID = 1L;

        @Override
        public void handleViolation(long elementTimestamp, long lastTimestamp) {}
    }

    /**
     * Handler that fails the program when timestamp monotony is violated.
     */
    public static final class FailingHandler implements MonotonyViolationHandler {
        private static final long serialVersionUID = 1L;

        @Override
        public void handleViolation(long elementTimestamp, long lastTimestamp) {
            throw new RuntimeException("Ascending timestamps condition violated. Element timestamp "
                    + elementTimestamp + " is smaller than last timestamp " + lastTimestamp);
        }
    }

    /**
     * Handler that only logs violations of timestamp monotony, on WARN log level.
     */
    public static final class LoggingHandler implements MonotonyViolationHandler {
        private static final long serialVersionUID = 1L;

        private static final Logger LOG = LoggerFactory.getLogger(AscendingTimestampExtractor.class);

        @Override
        public void handleViolation(long elementTimestamp, long lastTimestamp) {
            LOG.warn("Timestamp monotony violated: {} < {}", elementTimestamp, lastTimestamp);
        }
    }
  • Monotonyviolationhandler inherits serializable and defines the handleviolation method. This interface has three built-in implementation classes: ignorehandler, failinghandler and failinghandler
  • The handleviolation method of ignorehandler does not do any processing; The handleviolation of failinghandler will throw runtimeException; The logginghandler’s handleviolation method prints the warn log
  • Ascendingtimestampextractor uses logginghandler by default, which can also be set through withviolationhandler method

example

    @Test
    public void testWithFailingHandler() {
        AscendingTimestampExtractor<Long> extractor = (new AscendingTimestampExtractorTest.LongExtractor()).withViolationHandler(new FailingHandler());
        this.runValidTests(extractor);

        try {
            this.runInvalidTest(extractor);
            Assert.fail("should fail with an exception");
        } catch (Exception var3) {
            ;
        }

    }

    private void runValidTests(AscendingTimestampExtractor<Long> extractor) {
        Assert.assertEquals(13L, extractor.extractTimestamp(13L, -1L));
        Assert.assertEquals(13L, extractor.extractTimestamp(13L, 0L));
        Assert.assertEquals(14L, extractor.extractTimestamp(14L, 0L));
        Assert.assertEquals(20L, extractor.extractTimestamp(20L, 0L));
        Assert.assertEquals(20L, extractor.extractTimestamp(20L, 0L));
        Assert.assertEquals(20L, extractor.extractTimestamp(20L, 0L));
        Assert.assertEquals(500L, extractor.extractTimestamp(500L, 0L));
        Assert.assertEquals(9223372036854775806L, extractor.extractTimestamp(9223372036854775806L, 99999L));
    }

    private void runInvalidTest(AscendingTimestampExtractor<Long> extractor) {
        Assert.assertEquals(1000L, extractor.extractTimestamp(1000L, 100L));
        Assert.assertEquals(1000L, extractor.extractTimestamp(1000L, 100L));
        Assert.assertEquals(999L, extractor.extractTimestamp(999L, 100L));
    }

    private static class LongExtractor extends AscendingTimestampExtractor<Long> {
        private static final long serialVersionUID = 1L;

        private LongExtractor() {
        }

        public long extractAscendingTimestamp(Long element) {
            return element;
        }
    }
  • Here, withviolationhandler is used to set the violationhandler to failinghandler. When 999 is encountered, the monotonyviolationhandler.handleviiolation method will be called because it is smaller than the previous 1000

Summary

  • To facilitate development, flyk provides several built-in pre defined timestamp extractors / watermark emitters, one of which is ascendingtimestamp extractor
  • The ascendingtimestampextractor abstract class implements the extracttimestamp and getcurrentwatermark methods of the assignerwithperiodicwatermarks interface, and declares the abstract method extractascendingtimestamp for subclass implementation
  • Ascendingtimestampextractor is applicable to the time of elements, which is monotonically increasing in each parallel task. For the violation of timestamp monotony, the handleviolation method of monotonyviolationhandler is called here for processing; Monotonyviolationhandler inherits serializable and defines the handleviolation method. This interface has three built-in implementation classes: ignorehandler, failinghandler and failinghandler

doc

Recommended Today

Swift advanced (XV) extension

The extension in swift is somewhat similar to the category in OC Extension can beenumeration、structural morphology、class、agreementAdd new features□ you can add methods, calculation attributes, subscripts, (convenient) initializers, nested types, protocols, etc What extensions can’t do:□ original functions cannot be overwritten□ you cannot add storage attributes or add attribute observers to existing attributes□ cannot add parent […]